Nebula-storage : memory leak?

Users Issues & Bugs
1

Hi admin,

I wonder why is nebula-storage used a lot of memory? Is it leak memory ?

1
13296×1650 1.8 MB

After a restart, use a lot of CPU (a little memory RAM) :

2
23316×1212 1.16 MB
3
33308×870 808 KB

My env:

  • OS: uname -a => Linux touchee 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  • Compliler: g++ --version or clang++ --version => g++ (Ubuntu 9.3.0-11ubuntu0~18.04.1) 9.3.0
  • CPU: lscpu => 4 cores , Intel(R) Core™ i5-6400 CPU @ 2.70GHz
2

Q1: memory
There are some cache(LRU for example) in Nebula. How many partition you use? And, is the memory increase monotonic ?

Q2: cpu
Nebula storage use RocksDB, after restart, rocksdb will do compaction in mins to hours. CPU is high because of this.

3

Hi admin,

Q1: memory
I used the default config file (in nebula github)
When I created a new space, I used the default partition number and replica factor.

Q2: cpu
You said that the reason is RocksDB => I confuse because I used RocksDB in my project (deployed in server dev which nebula services were running) => my RocksDB is light (CPU), fast

Screen Shot 2020-08-21 at 20.49.47
Screen Shot 2020-08-21 at 20.49.472390×1578 404 KB

1 Like
4

Hi, linuxpham
Q1: mem
could you please paste your conf here? Looks like you run nebula on a desktop, and the default config is for much powerful server (40core, 200G+ mem), may be you need to adjust your configs.

Q2: CPU
Is storage CPU usage keeps high? In my opinion, after restart, it will goes high for about some minutes (worst case, hours), because of RocksDB compaction. And after the compaction, everything will goes fine.

1 Like
5

Hi admin,

This is nebula-storage conf file :

> Blockquote
########## basics ##########

# Whether to run as a daemon process

--daemonize=true

# The file to host the process id

--pid_file=/usr/local/nebula/pids/nebula-storaged.pid

########## logging ##########

# The directory to host logging files, which must already exists

--log_dir=/usr/local/nebula/logs

# Log level, 0, 1, 2, 3 for INFO, WARNING, ERROR, FATAL respectively

--minloglevel=0

# Verbose log level, 1, 2, 3, 4, the higher of the level, the more verbose of the logging

--v=0

# Maximum seconds to buffer the log messages

--logbufsecs=0

########## networking ##########

# Meta server address

--meta_server_addrs=127.0.0.1:45502

# Local ip

--local_ip=127.0.0.1

# Storage daemon listening port

--port=46500

# HTTP service ip

--ws_ip=127.0.0.1

# HTTP service port

--ws_http_port=12100

# HTTP2 service port

--ws_h2_port=12102

# heartbeat with meta service

--heartbeat_interval_secs=10

######### Raft #########

# Raft election timeout

--raft_heartbeat_interval_secs=30

# RPC timeout for raft client (ms)

--raft_rpc_timeout_ms=500

## recycle Raft WAL

--wal_ttl=3600

########## Disk ##########

# Root data path. Split by comma. e.g. --data_path=/disk1/path1/,/disk2/path2/

# One path per Rocksdb instance.

--data_path=/usr/local/nebula/data/storage

############## Rocksdb Options ##############

# The default reserved bytes for one batch operation

--rocksdb_batch_size=4096

# The default block cache size used in BlockBasedTable. (MB)

# recommend: 1/3 of all memory

--rocksdb_block_cache=4

# Compression algorithm, options: no,snappy,lz4,lz4hc,zlib,bzip2,zstd

# For the sake of binary compatibility, the default value is snappy.

# Recommend to use:

# * lz4 to gain more CPU performance, with the same compression ratio with snappy

# * zstd to occupy less disk space

# * lz4hc for the read-heavy write-light scenario

--rocksdb_compression=snappy

# Set different compressions for different levels

# For example, if --rocksdb_compression is snappy,

# "no:no:lz4:lz4::zstd" is identical to "no:no:lz4:lz4:snappy:zstd:snappy"

# In order to disable compression for level 0/1, set it to "no:no"

--rocksdb_compression_per_level=

--rocksdb_disable_wal=true

############## rocksdb Options ##############

--rocksdb_disable_wal=true

# rocksdb DBOptions in json, each name and value of option is a string, given as "option_name":"option_value" separated by comma

--rocksdb_db_options={"max_subcompactions":"1","max_background_jobs":"1"}

# rocksdb ColumnFamilyOptions in json, each name and value of option is string, given as "option_name":"option_value" separated by comma

--rocksdb_column_family_options={"disable_auto_compactions":"false","write_buffer_size":"67108864","max_write_buffer_number":"4","max_bytes_for_level_base":"268435456"}

# rocksdb BlockBasedTableOptions in json, each name and value of option is string, given as "option_name":"option_value" separated by comma

--rocksdb_block_based_table_options={"block_size":"8192"}

############# edge samplings ##############

# --enable_reservoir_sampling=false

# --max_edge_returned_per_vertex=2147483647
6

We have compilation options for memory detection，could you help turn on this compile macro and recompile?
cmake -DENABLE_ASAN=on ..

7

also , if you try nebula on a desktop, you may change your partition num to 5 or 10 (default is 100)

8

Hi linuxpham,

how is going on now ?

Let me explain a bit more about how nebula cache take memory:

for each partition, nebula take 16M for WAL cache.

also, it takes 1G for rocksdb block cache (for each space).

But this happens only when you inserting data.

If this isn’t the case, the only reason I can image now is

rocksdb is doing compaction. It may takes lots memory and CPU.

your may check nebula storage log if this printed

"Rocksdb compact column family: … "

9

Hi admin,

I found this problem.

A reason is: there are many spaces in nubela-graph

But i confuse: these spaces are empty (vertex and edge) => that means nebula-graph only takes a little memory ?

Thanks !

10

emmm… not get your point, could you please a bit more ?