Fix broker gc log options #11285

gaozhangmin · 2021-07-12T10:45:49Z

It seems that, We configured params for broker gc log, but not used.
turn on broker gc log.

Anonymitaet · 2021-07-13T01:56:47Z

Thanks for your contribution. For this PR, do we need to update docs?

gaozhangmin · 2021-07-13T03:01:07Z

Thanks for your contribution. For this PR, do we need to update docs?

I think there is no need to update docs

gaozhangmin · 2021-07-13T04:04:26Z

Anyone can review it?

gaozhangmin · 2021-09-24T03:34:17Z

/pulsarbot run-failure-checks

eolivelli

LGTM

@michaeljmarshall PTAL

gaozhangmin · 2021-09-24T14:59:53Z

/pulsarbot run-failure-checks

michaeljmarshall

Looks like a good change. Before we merge this, we should update the bookkeeper shell script and the Pulsar shell script to document the new environment variables.

pulsar/bin/bookkeeper

Lines 99 to 106 in 3b76784

    
           Environment variables: 
        
              BOOKIE_LOG_CONF        Log4j configuration file (default $DEFAULT_LOG_CONF) 
        
              BOOKIE_CONF            Configuration file (default: $DEFAULT_CONF) 
        
              BOOKIE_EXTRA_OPTS      Extra options to be passed to the jvm 
        
              BOOKIE_EXTRA_CLASSPATH Add extra paths to the bookkeeper classpath 
        
              ENTRY_FORMATTER_CLASS  Entry formatter class to format entries. 
        
              BOOKIE_PID_DIR         Folder where the Bookie server PID file should be stored 
        
              BOOKIE_STOP_TIMEOUT    Wait time before forcefully kill the Bookie server instance, if the stop is not successful

pulsar/bin/pulsar

Lines 158 to 173 in 3b76784

    
           Environment variables: 
        
              FUNCTIONS_LOG_CONF            Function Log4j configuration file (default $DEFAULT_FUNCTIONS_LOG_CONF) 
        
              PULSAR_LOG_CONF               Log4j configuration file (default $DEFAULT_LOG_CONF) 
        
              PULSAR_BROKER_CONF            Configuration file for broker (default: $DEFAULT_BROKER_CONF) 
        
              PULSAR_BOOKKEEPER_CONF        Configuration file for bookie (default: $DEFAULT_BOOKKEEPER_CONF) 
        
              PULSAR_ZK_CONF                Configuration file for zookeeper (default: $DEFAULT_ZK_CONF) 
        
              PULSAR_CONFIGURATION_STORE_CONF         Configuration file for global configuration store (default: $DEFAULT_CONFIGURATION_STORE_CONF) 
        
              PULSAR_WEBSOCKET_CONF         Configuration file for websocket proxy (default: $DEFAULT_WEBSOCKET_CONF) 
        
              PULSAR_PROXY_CONF             Configuration file for Pulsar proxy (default: $DEFAULT_PROXY_CONF) 
        
              PULSAR_WORKER_CONF            Configuration file for functions worker (default: $DEFAULT_WORKER_CONF) 
        
              PULSAR_STANDALONE_CONF        Configuration file for standalone (default: $DEFAULT_STANDALONE_CONF) 
        
              PULSAR_PRESTO_CONF            Configuration directory for Pulsar Presto (default: $DEFAULT_PULSAR_PRESTO_CONF) 
        
              PULSAR_EXTRA_OPTS             Extra options to be passed to the jvm 
        
              PULSAR_EXTRA_CLASSPATH        Add extra paths to the pulsar classpath 
        
              PULSAR_PID_DIR                Folder where the pulsar server PID file should be stored 
        
              PULSAR_STOP_TIMEOUT           Wait time before forcefully kill the pulsar server instance, if the stop is not successful

We also need to update the CLI tools web page: https://github.com/apache/pulsar/blob/master/site2/docs/reference-cli-tools.md

conf/bkenv.sh

gaozhangmin · 2021-09-26T03:17:58Z

test failed reason: Unrecognized option: -Xlog:gc:logs/pulsar_gc_%p.log:time,uptime:filecount=10,filesize=20M
I cannot figure out why.

bin/pulsar

michaeljmarshall · 2021-09-30T01:59:38Z

@gaozhangmin - I think you still need to update the documentation here: https://github.com/apache/pulsar/blob/master/site2/docs/reference-cli-tools.md.

gaozhangmin · 2021-09-30T06:36:36Z

@gaozhangmin - I think you still need to update the documentation here: https://github.com/apache/pulsar/blob/master/site2/docs/reference-cli-tools.md.

@michaeljmarshall DOCS added

BewareMyPower · 2021-10-14T08:29:40Z

@michaeljmarshall Could you take a look again?

BewareMyPower · 2021-10-19T02:38:28Z

Could you check my comments? It looks like pulsar-client-cpp/pulsar-test-service-start.sh cannot run successfully after this change.

gaozhangmin · 2021-10-19T02:52:38Z

Could you check my comments? It looks like pulsar-client-cpp/pulsar-test-service-start.sh cannot run successfully after this change.

@BewareMyPower Sure, I will check later.

gaozhangmin · 2021-10-19T08:09:21Z

Could you check my comments? It looks like pulsar-client-cpp/pulsar-test-service-start.sh cannot run successfully after this change.

@BewareMyPower Sure, I will check later.

@BewareMyPower I tested locally pulsar-client-cpp/pulsar-test-service-start.sh, successfully run without error.

gaozhangmin · 2021-10-19T08:12:24Z

/pulsarbot run-failure-checks

BewareMyPower · 2021-10-19T12:24:50Z

Could you check my comments? It looks like pulsar-client-cpp/pulsar-test-service-start.sh cannot run successfully after this change.

@BewareMyPower Sure, I will check later.

@BewareMyPower I tested locally pulsar-client-cpp/pulsar-test-service-start.sh, successfully run without error.

Did you use Java 11? The CI env is a docker container based on Ubuntu 16.04 and Java 11.

$ java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.16.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.16.04, mixed mode, sharing)

gaozhangmin · 2021-10-19T12:51:13Z

Could you check my comments? It looks like pulsar-client-cpp/pulsar-test-service-start.sh cannot run successfully after this change.

@BewareMyPower Sure, I will check later.

@BewareMyPower I tested locally pulsar-client-cpp/pulsar-test-service-start.sh, successfully run without error.

Did you use Java 11? The CI env is a docker container based on Ubuntu 16.04 and Java 11.
$ java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.16.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.16.04, mixed mode, sharing)

I tested without PULSAR_GC_LOG successfully, I don't know why this jdk11 cannot recognize xlog:gc

BewareMyPower · 2021-10-19T15:47:30Z

I will take a look at this issue tomorrow.

BewareMyPower · 2021-10-20T07:58:42Z

I found the problem. It's because when the standalone starts, it chooses the /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java rather than /usr/bin/java.

$ /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)
$ java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.16.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.16.04, mixed mode, sharing)

I think it's caused by wrong JAVA_HOME variable.

BewareMyPower · 2021-10-20T08:04:30Z

You can modify pulsar-client-cpp/run-unit-tests.sh line 26 from

./pulsar-test-service-start.sh

to

JAVA_HOME=/usr ./pulsar-test-service-start.sh

Then bin/pulsar would choose /usr/bin/java rather than the original $JAVA_HOME/bin/java.

gaozhangmin · 2021-10-20T08:42:37Z

/pulsarbot run-failure-checks

gaozhangmin · 2021-10-20T09:06:11Z

You can modify pulsar-client-cpp/run-unit-tests.sh line 26 from
./pulsar-test-service-start.sh
to
JAVA_HOME=/usr ./pulsar-test-service-start.sh 
Then bin/pulsar would choose /usr/bin/java rather than the original $JAVA_HOME/bin/java.

Thx. It worked.

michaeljmarshall

LGTM

gaozhangmin · 2021-11-23T02:32:55Z

@codelipenghui PTAL again

lhotari · 2021-11-29T10:48:20Z

conf/pulsar_env.sh


 # Extra options to be passed to the jvm
-PULSAR_EXTRA_OPTS=${PULSAR_EXTRA_OPTS:-" -Dpulsar.allocator.exit_on_oom=true -Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024"}
+PULSAR_EXTRA_OPTS="${PULSAR_EXTRA_OPTS:-" -Dpulsar.allocator.exit_on_oom=true -Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024"} ${PULSAR_GC_LOG}"


This change seems to break pulsar-perf when running it in the docker container.

example failure:

root@e187af4bc610:/pulsar# ./bin/pulsar-perf produce [0.000s][error][logging] Error opening log file 'logs/pulsar_gc_119.log': No such file or directory [0.000s][error][logging] Initialization of output 'file=logs/pulsar_gc_%p.log' using options 'filecount=10,filesize=20M' failed. Invalid -Xlog option '-Xlog:gc*:logs/pulsar_gc_%p.log:time,uptime:filecount=10,filesize=20M', see error log for details. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.

I created #13022 with a minimal fix to the issue.

- create /pulsar/logs directory since it's now required for running pulsar-perf after apache#11285 changes

Anonymitaet added the doc-not-needed Your PR changes do not impact docs label Jul 13, 2021

sijie approved these changes Jul 14, 2021

View reviewed changes

sijie assigned gaozhangmin Jul 14, 2021

sijie added this to the 2.9.0 milestone Jul 14, 2021

sijie requested review from addisonj, codelipenghui, hangc0276 and merlimat July 14, 2021 06:44

gaozhangmin force-pushed the broker_gc_log branch from 5eb639f to d29c059 Compare September 24, 2021 02:21

eolivelli approved these changes Sep 24, 2021

View reviewed changes

gaozhangmin force-pushed the broker_gc_log branch from 0c23420 to 44d94d2 Compare September 24, 2021 08:44

michaeljmarshall requested changes Sep 24, 2021

View reviewed changes

conf/bkenv.sh Outdated Show resolved Hide resolved

gaozhangmin force-pushed the broker_gc_log branch from bbca224 to 38ce3ce Compare September 26, 2021 02:30

gaozhangmin force-pushed the broker_gc_log branch 2 times, most recently from 38ce3ce to ce012fe Compare September 26, 2021 12:23

gaozhangmin requested review from eolivelli and sijie September 26, 2021 12:24

gaozhangmin commented Sep 26, 2021

View reviewed changes

bin/pulsar Show resolved Hide resolved

eolivelli modified the milestones: 2.9.0, 2.10.0 Oct 6, 2021

BewareMyPower approved these changes Oct 14, 2021

View reviewed changes

gaozhangmin force-pushed the broker_gc_log branch 3 times, most recently from 966c4e2 to d0d389b Compare October 19, 2021 11:15

gaozhangmin force-pushed the broker_gc_log branch from 27fdfb3 to d0d389b Compare October 19, 2021 12:50

broker gc log

a79a123

gaozhangmin force-pushed the broker_gc_log branch from d0d389b to a79a123 Compare October 20, 2021 05:40

fix cpp test failed

34ff47d

michaeljmarshall approved these changes Oct 25, 2021

View reviewed changes

codelipenghui approved these changes Nov 24, 2021

View reviewed changes

codelipenghui merged commit c94c0aa into apache:master Nov 24, 2021

codelipenghui changed the title ~~broker gc log~~ Fix broker gc log options Nov 24, 2021

eolivelli pushed a commit to eolivelli/pulsar that referenced this pull request Nov 29, 2021

Fix broker gc log options (apache#11285)

dfcf916

lhotari reviewed Nov 29, 2021

View reviewed changes

lhotari added a commit to lhotari/pulsar that referenced this pull request Nov 29, 2021

Fix running pulsar-perf in docker image after apache#11285 changes

b0b2baf

- create /pulsar/logs directory since it's now required for running pulsar-perf after apache#11285 changes

This was referenced Nov 29, 2021

Fix running pulsar-perf in docker image after #11285 changes #13022

Closed

Fix usage of PULSAR_EXTRA_OPTS/BOOKIE_EXTRA_OPTS in startup scripts #13025

Merged

MMirelli mentioned this pull request Dec 6, 2021

Fix initialisation of autorecovery pod #13155

Merged

2 tasks

fxbing pushed a commit to fxbing/pulsar that referenced this pull request Dec 19, 2021

Fix broker gc log options (apache#11285)

7cc69aa

	Environment variables:
	BOOKIE_LOG_CONF Log4j configuration file (default $DEFAULT_LOG_CONF)
	BOOKIE_CONF Configuration file (default: $DEFAULT_CONF)
	BOOKIE_EXTRA_OPTS Extra options to be passed to the jvm
	BOOKIE_EXTRA_CLASSPATH Add extra paths to the bookkeeper classpath
	ENTRY_FORMATTER_CLASS Entry formatter class to format entries.
	BOOKIE_PID_DIR Folder where the Bookie server PID file should be stored
	BOOKIE_STOP_TIMEOUT Wait time before forcefully kill the Bookie server instance, if the stop is not successful

	Environment variables:
	FUNCTIONS_LOG_CONF Function Log4j configuration file (default $DEFAULT_FUNCTIONS_LOG_CONF)
	PULSAR_LOG_CONF Log4j configuration file (default $DEFAULT_LOG_CONF)
	PULSAR_BROKER_CONF Configuration file for broker (default: $DEFAULT_BROKER_CONF)
	PULSAR_BOOKKEEPER_CONF Configuration file for bookie (default: $DEFAULT_BOOKKEEPER_CONF)
	PULSAR_ZK_CONF Configuration file for zookeeper (default: $DEFAULT_ZK_CONF)
	PULSAR_CONFIGURATION_STORE_CONF Configuration file for global configuration store (default: $DEFAULT_CONFIGURATION_STORE_CONF)
	PULSAR_WEBSOCKET_CONF Configuration file for websocket proxy (default: $DEFAULT_WEBSOCKET_CONF)
	PULSAR_PROXY_CONF Configuration file for Pulsar proxy (default: $DEFAULT_PROXY_CONF)
	PULSAR_WORKER_CONF Configuration file for functions worker (default: $DEFAULT_WORKER_CONF)
	PULSAR_STANDALONE_CONF Configuration file for standalone (default: $DEFAULT_STANDALONE_CONF)
	PULSAR_PRESTO_CONF Configuration directory for Pulsar Presto (default: $DEFAULT_PULSAR_PRESTO_CONF)
	PULSAR_EXTRA_OPTS Extra options to be passed to the jvm
	PULSAR_EXTRA_CLASSPATH Add extra paths to the pulsar classpath
	PULSAR_PID_DIR Folder where the pulsar server PID file should be stored
	PULSAR_STOP_TIMEOUT Wait time before forcefully kill the pulsar server instance, if the stop is not successful

Fix broker gc log options #11285

Fix broker gc log options #11285

Uh oh!

Conversation

gaozhangmin commented Jul 12, 2021

Uh oh!

Anonymitaet commented Jul 13, 2021

Uh oh!

gaozhangmin commented Jul 13, 2021

Uh oh!

gaozhangmin commented Jul 13, 2021

Uh oh!

gaozhangmin commented Sep 24, 2021

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

gaozhangmin commented Sep 24, 2021

Uh oh!

michaeljmarshall left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gaozhangmin commented Sep 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

michaeljmarshall commented Sep 30, 2021

Uh oh!

gaozhangmin commented Sep 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BewareMyPower commented Oct 14, 2021

Uh oh!

BewareMyPower commented Oct 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaozhangmin commented Oct 19, 2021

Uh oh!

gaozhangmin commented Oct 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaozhangmin commented Oct 19, 2021

Uh oh!

BewareMyPower commented Oct 19, 2021

Uh oh!

gaozhangmin commented Oct 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BewareMyPower commented Oct 19, 2021

Uh oh!

BewareMyPower commented Oct 20, 2021

Uh oh!

BewareMyPower commented Oct 20, 2021

Uh oh!

gaozhangmin commented Oct 20, 2021

Uh oh!

gaozhangmin commented Oct 20, 2021

Uh oh!

michaeljmarshall left a comment

Choose a reason for hiding this comment

Uh oh!

gaozhangmin commented Nov 23, 2021

Uh oh!

lhotari Nov 29, 2021

Choose a reason for hiding this comment

Uh oh!

lhotari Nov 29, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

michaeljmarshall left a comment •

edited

Loading

gaozhangmin commented Sep 26, 2021 •

edited

Loading

gaozhangmin commented Sep 30, 2021 •

edited

Loading

BewareMyPower commented Oct 19, 2021 •

edited

Loading

gaozhangmin commented Oct 19, 2021 •

edited

Loading

gaozhangmin commented Oct 19, 2021 •

edited

Loading