Re: [PATCH 1/2] pmacct: New addon

18 Mar 2021

Hello,
I will split this email into two parts:
The technical part:
* The initscript isn’t based on our template and uses sleep to check if starting a daemon has been successful. That should not be necessary and will potentially slow down the boot process.
* I am not very sure if the example configuration files are helpful. Maybe someone can clear this up a little bit for me what is the rationale is behind them.
* There are loads of daemons being shipped and only one is being used.
* How much load does this generate? How large is the captured data?
Then there is the other part:
...
On 8 Mar 2021, at 11:57, Adolf Belka (ipfire) adolf.belka@ipfire.org wrote:
From: Adolf Belka adolf.belka@ipfire.org

What is it?
pmacct is a monitoring tool for network management tasks. Data collected
can be used for analysis and troubleshooting purposes to maintain the
health of the network. pmacct can collect, replicate and export network
information. It can cache in memory tables, store persistently to SQLite3
and output to flat-files like CSV, formatted, and JSON.

Okay.
...

Why is it needed?
To monitor data usage (IP-based or MAC-based data accounting) down to the
client level. Net-Traffic will monitor traffic for the entire RED, GREEN,
etc. networks, but it cannot pinpoint which client is using lots of data.
Connections will take a snapshot but not show day by day sums. pmacct can
help admins keep tabs on users that use too much data.

I agree with the goal of it. But how does this add-on achieve that?
The data is being written to a (usually binary) file which cannot be visualised in the web UI. Does this not want a CGI script that allows people to see what is happening?
Sending manual SQL queries probably isn’t much fun :)
...

What are the use cases?
An ISP may implement data caps and if the limit is over-run then you have
to pay for every additional xxGB of data used. Typical charges can be
around $10 per 50GB. With pmacct you can identify the high users and take
action, hopefully before the limit is breached.
This is being introduced as a command line only tool. However, at a later
date, if it is useful to enough additional users a WUI page could be
developed as discussed in the development mailing list
https://lists.ipfire.org/pipermail/development/2021-January/009174.html

I have been thinking about this for a while now and would say that I can approve merging this as a CLI tool. But it would not really enhance IPFire’s functionality because it is only usable for people who spend a lot of time writing their own scripts or SQL queries to get information that can be used to make decisions.
Adding another alternative to vnstat will result in us having to maintain two things that roughly do the same - or where one’s features are a subset of the other one.
I would like to avoid merging this and then see it abandoned because it is too complicated to use. We already have a lot of virtually unmaintained stuff around and I would really like to drop a lot of it, although I know that there are a few people actually using it.
Thoughts?
Best,
-Michael
...
Co-authored-by: Jon Murphy
Co-authored-by: Erik Kapfer ummeegge@ipfire.org
Signed-off-by: Adolf Belka adolf.belka@ipfire.org
config/backup/includes/pmacct             |   2 +
config/pmacct/pmacct.conf                 |  24 +++++
config/pmacct/pmacct.init                 |  57 +++++++++++
config/pmacct/pmacct_memory_example.conf  |  24 +++++
config/pmacct/pmacct_sqlite3_example.conf |  25 +++++
config/rootfiles/packages/pmacct          | 111 ++++++++++++++++++++++
lfs/pmacct                                | 107 +++++++++++++++++++++
make.sh                                   |   2 +
src/paks/pmacct/install.sh                |  36 +++++++
src/paks/pmacct/uninstall.sh              |  33 +++++++
src/paks/pmacct/update.sh                 |  26 +++++
11 files changed, 447 insertions(+)
create mode 100644 config/backup/includes/pmacct
create mode 100644 config/pmacct/pmacct.conf
create mode 100644 config/pmacct/pmacct.init
create mode 100644 config/pmacct/pmacct_memory_example.conf
create mode 100644 config/pmacct/pmacct_sqlite3_example.conf
create mode 100644 config/rootfiles/packages/pmacct
create mode 100644 lfs/pmacct
create mode 100644 src/paks/pmacct/install.sh
create mode 100644 src/paks/pmacct/uninstall.sh
create mode 100644 src/paks/pmacct/update.sh

diff --git a/config/backup/includes/pmacct b/config/backup/includes/pmacct
new file mode 100644
index 000000000..4359067ad
--- /dev/null
+++ b/config/backup/includes/pmacct
@@ -0,0 +1,2 @@
+/etc/pmacct/
+/var/spool/pmacct/
diff --git a/config/pmacct/pmacct.conf b/config/pmacct/pmacct.conf
new file mode 100644
index 000000000..3c1d47efa
--- /dev/null
+++ b/config/pmacct/pmacct.conf
@@ -0,0 +1,24 @@
+!
+! Pmacctd configuration file for IPFire environment
+!



+syslog: daemon
+daemonize: true
+debug: false
+promisc: true
+interface: green0




+!
+! "plugin1" plugin configuration
+!
+plugins: memory[plugin1]



+plugin_buffer_size[plugin1]: 102400
+plugin_pipe_size[plugin1]: 10240000



+imt_mem_pools_number: 256
+imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe



+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto
+aggregate_filter[plugin1]: ip
\ No newline at end of file
diff --git a/config/pmacct/pmacct.init b/config/pmacct/pmacct.init
new file mode 100644
index 000000000..31f9ce49b
--- /dev/null
+++ b/config/pmacct/pmacct.init
@@ -0,0 +1,57 @@
+#!/bin/bash



+# Begin $rc_base/init.d/pmacct



+# Starts and stops pmacct daemon
+# Date: 2021-02-25 12:25:30 (Thu, 25 Feb 2021)



+# Locations
+CONF="/etc/pmacct/pmacct.conf"
+CONTENT="/var/pmacct"
+# Pmacctd binary
+PMACCT="/usr/sbin/pmacctd"
+# Pmacctd start options '-f' for the configuration file 
+OPT="-f ${CONF}"
+DESC="pmacct daemon"
+# Pmacctd syslog facilitiy
+#SYSLOG="local1"




+. /etc/sysconfig/rc
+. $rc_functions



+case "$1" in

start)
boot_mesg "Starting the ${DESC}... "


loadproc ${PMACCT} ${OPT}


sleep 2


if P=$(pgrep -fl pmacctd); then


	echo -e "\n${P}"


else


	echo -e "\n${PMACCT} is not running... "


fi


;;



stop)
boot_mesg "Stopping the ${DESC}... "


killproc ${PMACCT}


;;



restart)
$0 stop


sleep 1


$0 start


;;



status)
statusproc ${PMACCT}


;;



*)
echo "Usage: $0 {start|stop|restart|status}"


exit 1


;;



+esac



+# End $rc_base/init.d/pmacct



diff --git a/config/pmacct/pmacct_memory_example.conf b/config/pmacct/pmacct_memory_example.conf
new file mode 100644
index 000000000..0babdaaf7
--- /dev/null
+++ b/config/pmacct/pmacct_memory_example.conf
@@ -0,0 +1,24 @@
+!
+! Pmacctd configuration file for IPFire environment
+!



+syslog: daemon
+daemonize: true
+debug: false
+promisc: true
+interface: green0




+!
+! "plugin1" plugin configuration
+!
+plugins: memory[plugin1]



+plugin_buffer_size[plugin1]: 102400
+plugin_pipe_size[plugin1]: 10240000



+imt_mem_pools_number: 256
+imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe



+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto
+aggregate_filter[plugin1]: ip
diff --git a/config/pmacct/pmacct_sqlite3_example.conf b/config/pmacct/pmacct_sqlite3_example.conf
new file mode 100644
index 000000000..7fb996408
--- /dev/null
+++ b/config/pmacct/pmacct_sqlite3_example.conf
@@ -0,0 +1,25 @@
+!
+! Pmacctd configuration file for IPFire environment
+!



+syslog: daemon
+daemonize: true
+debug: false
+promisc: true
+interface: green0




+!
+! "plugin2" plugin configuration
+!
+plugins: sqlite3[plugin2]



+sql_db[plugin2]: /var/spool/pmacct/pmacct_sqlitev1.db
+sql_table_version[plugin2]: 1
+sql_history[plugin2]: 5m
+sql_refresh_time[plugin2]: 300
+sql_history_roundoff[plugin2]: m
+sql_dont_try_update: true



+aggregate[plugin2]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto
+aggregate_filter[plugin2]: ip
diff --git a/config/rootfiles/packages/pmacct b/config/rootfiles/packages/pmacct
new file mode 100644
index 000000000..fed83ee67
--- /dev/null
+++ b/config/rootfiles/packages/pmacct
@@ -0,0 +1,111 @@
+etc/pmacct
+etc/pmacct/pmacct.conf
+etc/pmacct/pmacct_memory_example.conf
+etc/pmacct/pmacct_sqlite3_example.conf
+etc/rc.d/init.d/pmacct
+usr/bin/pmacct
+#usr/lib/pmacct
+#usr/lib/pmacct/examples
+#usr/lib/pmacct/examples/custom
+#usr/lib/pmacct/examples/custom/libcustom.la
+#usr/lib/pmacct/examples/lg
+#usr/lib/pmacct/examples/lg/pmbgp
+usr/sbin/nfacctd
+usr/sbin/pmacctd
+usr/sbin/pmbgpd
+usr/sbin/pmbmpd
+usr/sbin/pmtelemetryd
+usr/sbin/sfacctd
+usr/share/pmacct
+#usr/share/pmacct/CONFIG-KEYS
+#usr/share/pmacct/FAQS
+#usr/share/pmacct/QUICKSTART
+#usr/share/pmacct/UPGRADE
+#usr/share/pmacct/docs
+#usr/share/pmacct/docs/IPFIX
+#usr/share/pmacct/docs/LOOKING_GLASS_FORMAT
+#usr/share/pmacct/docs/MSGLOG_DUMP_FORMATS
+#usr/share/pmacct/docs/SIGNALS
+#usr/share/pmacct/docs/TRIGGER_VARS
+#usr/share/pmacct/examples
+usr/share/pmacct/examples/allow.lst.example
+#usr/share/pmacct/examples/amqp
+usr/share/pmacct/examples/amqp/amqp_receiver.py
+#usr/share/pmacct/examples/avro
+usr/share/pmacct/examples/avro/avro_file_decoder.py
+usr/share/pmacct/examples/bgp_agent.map.example
+usr/share/pmacct/examples/bgp_md5.lst.example
+usr/share/pmacct/examples/bgp_xconnects.map.example
+usr/share/pmacct/examples/flow_to_rd.map.example
+#usr/share/pmacct/examples/kafka
+usr/share/pmacct/examples/kafka/kafka_consumer.py
+#usr/share/pmacct/examples/lg
+usr/share/pmacct/examples/lg/pmbgp.py
+usr/share/pmacct/examples/networks.lst.example
+usr/share/pmacct/examples/nfacctd-sql.conf.example
+usr/share/pmacct/examples/pcap_interfaces.map.example
+usr/share/pmacct/examples/peers.map.example
+usr/share/pmacct/examples/pmacctd-imt.conf.example
+usr/share/pmacct/examples/pmacctd-multiple-plugins.conf.example
+usr/share/pmacct/examples/pmacctd-sql.conf.example
+usr/share/pmacct/examples/pmacctd-sqlite3.conf.example
+usr/share/pmacct/examples/ports.lst.example
+usr/share/pmacct/examples/pretag.map.example
+usr/share/pmacct/examples/primitives.lst.example
+usr/share/pmacct/examples/probe_netflow.conf.example
+usr/share/pmacct/examples/probe_sflow.conf.example
+usr/share/pmacct/examples/sampling.map.example
+usr/share/pmacct/examples/tee_receivers.lst.example
+#usr/share/pmacct/sql
+#usr/share/pmacct/sql/README.64bit
+#usr/share/pmacct/sql/README.GeoIP
+#usr/share/pmacct/sql/README.IPv6
+#usr/share/pmacct/sql/README.cos
+#usr/share/pmacct/sql/README.custom_primitives
+#usr/share/pmacct/sql/README.etype
+#usr/share/pmacct/sql/README.export_proto
+#usr/share/pmacct/sql/README.iface
+#usr/share/pmacct/sql/README.label
+#usr/share/pmacct/sql/README.mask
+#usr/share/pmacct/sql/README.mpls
+#usr/share/pmacct/sql/README.mysql
+#usr/share/pmacct/sql/README.nat
+#usr/share/pmacct/sql/README.pgsql
+#usr/share/pmacct/sql/README.sampling
+usr/share/pmacct/sql/README.sqlite3
+#usr/share/pmacct/sql/README.tag2
+#usr/share/pmacct/sql/README.timestamp
+#usr/share/pmacct/sql/README.tunnel
+#usr/share/pmacct/sql/pmacct-create-db.pgsql
+#usr/share/pmacct/sql/pmacct-create-db_bgp_v1.mysql
+#usr/share/pmacct/sql/pmacct-create-db_v1.mysql
+#usr/share/pmacct/sql/pmacct-create-db_v2.mysql
+#usr/share/pmacct/sql/pmacct-create-db_v3.mysql
+#usr/share/pmacct/sql/pmacct-create-db_v4.mysql
+#usr/share/pmacct/sql/pmacct-create-db_v5.mysql
+#usr/share/pmacct/sql/pmacct-create-db_v6.mysql
+#usr/share/pmacct/sql/pmacct-create-db_v7.mysql
+#usr/share/pmacct/sql/pmacct-create-db_v8.mysql
+#usr/share/pmacct/sql/pmacct-create-db_v9.mysql
+#usr/share/pmacct/sql/pmacct-create-table_bgp_v1.pgsql
+usr/share/pmacct/sql/pmacct-create-table_bgp_v1.sqlite3
+#usr/share/pmacct/sql/pmacct-create-table_v1.pgsql
+usr/share/pmacct/sql/pmacct-create-table_v1.sqlite3
+#usr/share/pmacct/sql/pmacct-create-table_v2.pgsql
+usr/share/pmacct/sql/pmacct-create-table_v2.sqlite3
+#usr/share/pmacct/sql/pmacct-create-table_v3.pgsql
+usr/share/pmacct/sql/pmacct-create-table_v3.sqlite3
+#usr/share/pmacct/sql/pmacct-create-table_v4.pgsql
+usr/share/pmacct/sql/pmacct-create-table_v4.sqlite3
+#usr/share/pmacct/sql/pmacct-create-table_v5.pgsql
+usr/share/pmacct/sql/pmacct-create-table_v5.sqlite3
+#usr/share/pmacct/sql/pmacct-create-table_v6.pgsql
+usr/share/pmacct/sql/pmacct-create-table_v6.sqlite3
+usr/share/pmacct/sql/pmacct-create-table_v7.sqlite3
+#usr/share/pmacct/sql/pmacct-create-table_v7_v8.pgsql
+usr/share/pmacct/sql/pmacct-create-table_v8.sqlite3
+#usr/share/pmacct/sql/pmacct-create-table_v9.pgsql
+usr/share/pmacct/sql/pmacct-create-table_v9.sqlite3
+#usr/share/pmacct/sql/pmacct-grant-db.mysql
+var/ipfire/backup/addons/includes/pmacct
+var/spool/pmacct
diff --git a/lfs/pmacct b/lfs/pmacct
new file mode 100644
index 000000000..c6cc5270e
--- /dev/null
+++ b/lfs/pmacct
@@ -0,0 +1,107 @@
+###############################################################################
+#                                                                             #
+# IPFire.org - A linux based firewall                                         #
+# Copyright (C) 2019  IPFire Team  info@ipfire.org                          #
+#                                                                             #
+# This program is free software: you can redistribute it and/or modify        #
+# it under the terms of the GNU General Public License as published by        #
+# the Free Software Foundation, either version 3 of the License, or           #
+# (at your option) any later version.                                         #
+#                                                                             #
+# This program is distributed in the hope that it will be useful,             #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
+# GNU General Public License for more details.                                #
+#                                                                             #
+# You should have received a copy of the GNU General Public License           #
+# along with this program.  If not, see http://www.gnu.org/licenses/.       #
+#                                                                             #
+###############################################################################



+###############################################################################
+# Definitions
+###############################################################################



+include Config



+VER        = 1.7.6



+THISAPP    = pmacct-$(VER)
+DL_FILE    = $(THISAPP).tar.gz
+DL_FROM    = $(URL_IPFIRE)
+DIR_APP    = $(DIR_SRC)/$(THISAPP)
+TARGET     = $(DIR_INFO)/$(THISAPP)
+PROG       = pmacct
+PAK_VER    = 1



+DEPS       = libcdada



+###############################################################################
+# Top-level Rules
+###############################################################################



+objects = $(DL_FILE)



+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)



+$(DL_FILE)_MD5 = b829cfdc394d5acd6d32fb23de181a6b



+install : $(TARGET)



+check : $(patsubst %,$(DIR_CHK)/%,$(objects))



+download :$(patsubst %,$(DIR_DL)/%,$(objects))



+md5 : $(subst %,%_MD5,$(objects))



+dist:

@$(PAK)


+###############################################################################
+# Downloading, checking, md5sum
+###############################################################################



+$(patsubst %,$(DIR_CHK)/%,$(objects)) :

@$(CHECK)


+$(patsubst %,$(DIR_DL)/%,$(objects)) :

@$(LOAD)


+$(subst %,%_MD5,$(objects)) :

@$(MD5)


+###############################################################################
+# Installation Details
+###############################################################################



+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))

@$(PREBUILD)
@rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE)
cd $(DIR_APP) && ./autogen.sh
cd $(DIR_APP) && ./configure 			\
--prefix=/usr 				\


--sysconfdir=/etc/pmacct 		\


--enable-sqlite3 			\


--enable-l2				\


--enable-jansson



# Add directory for plugins and pipes
-mkdir -v /var/spool/pmacct

# Add configuration folder and copy conf files to it
-mkdir -v /etc/pmacct
cp -vf $(DIR_CONF)/pmacct/*.conf /etc/pmacct

# Copy initscript to dir
cp -vf $(DIR_CONF)/pmacct/pmacct.init /etc/rc.d/init.d/pmacct
chmod 754 /etc/rc.d/init.d/pmacct
chown root:root /etc/rc.d/init.d/pmacct

cd $(DIR_APP) && make $(MAKETUNING)
cd $(DIR_APP) && make install

# Install backup definition
install -v -m 644 ${DIR_SRC}/config/backup/includes/pmacct \
	/var/ipfire/backup/addons/includes/pmacct


@rm -rf $(DIR_APP)
@$(POSTBUILD)

diff --git a/make.sh b/make.sh
index 43cccaca0..d42c9b1e2 100755
--- a/make.sh
+++ b/make.sh
@@ -1646,6 +1646,8 @@ buildipfire() {
  lfsmake2 ncdu
  lfsmake2 lshw
  lfsmake2 socat

lfsmake2 libcdada
lfsmake2 pmacct

}
buildinstaller() {
diff --git a/src/paks/pmacct/install.sh b/src/paks/pmacct/install.sh
new file mode 100644
index 000000000..8178cc7bc
--- /dev/null
+++ b/src/paks/pmacct/install.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+############################################################################
+#                                                                          #
+# This file is part of the IPFire Firewall.                                #
+#                                                                          #
+# IPFire is free software; you can redistribute it and/or modify           #
+# it under the terms of the GNU General Public License as published by     #
+# the Free Software Foundation; either version 2 of the License, or        #
+# (at your option) any later version.                                      #
+#                                                                          #
+# IPFire is distributed in the hope that it will be useful,                #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of           #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the            #
+# GNU General Public License for more details.                             #
+#                                                                          #
+# You should have received a copy of the GNU General Public License        #
+# along with IPFire; if not, write to the Free Software                    #
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA #
+#                                                                          #
+# Copyright (C) 2007 IPFire-Team info@ipfire.org.                        #
+#                                                                          #
+############################################################################
+#
+. /opt/pakfire/lib/functions.sh



+extract_files
+restore_backup ${NAME}



+# Add symlinks for runlevels
+ln -s ../init.d/${NAME} /etc/rc.d/rc0.d/K85${NAME}
+ln -s ../init.d/${NAME} /etc/rc.d/rc3.d/S50${NAME}
+ln -s ../init.d/${NAME} /etc/rc.d/rc6.d/K85${NAME}
+start_service --background ${NAME}




+# EOF
diff --git a/src/paks/pmacct/uninstall.sh b/src/paks/pmacct/uninstall.sh
new file mode 100644
index 000000000..51631e8c1
--- /dev/null
+++ b/src/paks/pmacct/uninstall.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+############################################################################
+#                                                                          #
+# This file is part of the IPFire Firewall.                                #
+#                                                                          #
+# IPFire is free software; you can redistribute it and/or modify           #
+# it under the terms of the GNU General Public License as published by     #
+# the Free Software Foundation; either version 2 of the License, or        #
+# (at your option) any later version.                                      #
+#                                                                          #
+# IPFire is distributed in the hope that it will be useful,                #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of           #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the            #
+# GNU General Public License for more details.                             #
+#                                                                          #
+# You should have received a copy of the GNU General Public License        #
+# along with IPFire; if not, write to the Free Software                    #
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA #
+#                                                                          #
+# Copyright (C) 2007 IPFire-Team info@ipfire.org.                        #
+#                                                                          #
+############################################################################
+#
+. /opt/pakfire/lib/functions.sh



+stop_service ${NAME}
+make_backup ${NAME}
+remove_files



+# Delete symlinks
+rm -rfv /etc/rc.d/rc?.d/???${NAME}



+# EOF
diff --git a/src/paks/pmacct/update.sh b/src/paks/pmacct/update.sh
new file mode 100644
index 000000000..89c40d0d7
--- /dev/null
+++ b/src/paks/pmacct/update.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+############################################################################
+#                                                                          #
+# This file is part of the IPFire Firewall.                                #
+#                                                                          #
+# IPFire is free software; you can redistribute it and/or modify           #
+# it under the terms of the GNU General Public License as published by     #
+# the Free Software Foundation; either version 2 of the License, or        #
+# (at your option) any later version.                                      #
+#                                                                          #
+# IPFire is distributed in the hope that it will be useful,                #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of           #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the            #
+# GNU General Public License for more details.                             #
+#                                                                          #
+# You should have received a copy of the GNU General Public License        #
+# along with IPFire; if not, write to the Free Software                    #
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA #
+#                                                                          #
+# Copyright (C) 2007 IPFire-Team info@ipfire.org.                        #
+#                                                                          #
+############################################################################
+#
+. /opt/pakfire/lib/functions.sh
+./uninstall.sh
+./install.sh
2.30.1

    

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Re: [PATCH 1/2] pmacct: New addon

Signed-off-by: Adolf Belka adolf.belka@ipfire.org

+./install.sh