From: Adolf Belka adolf.belka@ipfire.org
- What is it? pmacct is a monitoring tool for network management tasks. Data collected can be used for analysis and troubleshooting purposes to maintain the health of the network. pmacct can collect, replicate and export network information. It can cache in memory tables, store persistently to SQLite3 and output to flat-files like CSV, formatted, and JSON. - Why is it needed? To monitor data usage (IP-based or MAC-based data accounting) down to the client level. Net-Traffic will monitor traffic for the entire RED, GREEN, etc. networks, but it cannot pinpoint which client is using lots of data. Connections will take a snapshot but not show day by day sums. pmacct can help admins keep tabs on users that use too much data. - What are the use cases? An ISP may implement data caps and if the limit is over-run then you have to pay for every additional xxGB of data used. Typical charges can be around $10 per 50GB. With pmacct you can identify the high users and take action, hopefully before the limit is breached. - This is being introduced as a command line only tool. However, at a later date, if it is useful to enough additional users a WUI page could be developed as discussed in the development mailing list https://lists.ipfire.org/pipermail/development/2021-January/009174.html Co-authored-by: Jon Murphy Co-authored-by: Erik Kapfer ummeegge@ipfire.org
Signed-off-by: Adolf Belka adolf.belka@ipfire.org --- config/backup/includes/pmacct | 2 + config/pmacct/pmacct.conf | 24 +++++ config/pmacct/pmacct.init | 57 +++++++++++ config/pmacct/pmacct_memory_example.conf | 24 +++++ config/pmacct/pmacct_sqlite3_example.conf | 25 +++++ config/rootfiles/packages/pmacct | 111 ++++++++++++++++++++++ lfs/pmacct | 107 +++++++++++++++++++++ make.sh | 2 + src/paks/pmacct/install.sh | 36 +++++++ src/paks/pmacct/uninstall.sh | 33 +++++++ src/paks/pmacct/update.sh | 26 +++++ 11 files changed, 447 insertions(+) create mode 100644 config/backup/includes/pmacct create mode 100644 config/pmacct/pmacct.conf create mode 100644 config/pmacct/pmacct.init create mode 100644 config/pmacct/pmacct_memory_example.conf create mode 100644 config/pmacct/pmacct_sqlite3_example.conf create mode 100644 config/rootfiles/packages/pmacct create mode 100644 lfs/pmacct create mode 100644 src/paks/pmacct/install.sh create mode 100644 src/paks/pmacct/uninstall.sh create mode 100644 src/paks/pmacct/update.sh
diff --git a/config/backup/includes/pmacct b/config/backup/includes/pmacct new file mode 100644 index 000000000..4359067ad --- /dev/null +++ b/config/backup/includes/pmacct @@ -0,0 +1,2 @@ +/etc/pmacct/ +/var/spool/pmacct/ diff --git a/config/pmacct/pmacct.conf b/config/pmacct/pmacct.conf new file mode 100644 index 000000000..3c1d47efa --- /dev/null +++ b/config/pmacct/pmacct.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +! + +syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0 + + +! +! "plugin1" plugin configuration +! +plugins: memory[plugin1] + +plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000 + +imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe + +aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip \ No newline at end of file diff --git a/config/pmacct/pmacct.init b/config/pmacct/pmacct.init new file mode 100644 index 000000000..31f9ce49b --- /dev/null +++ b/config/pmacct/pmacct.init @@ -0,0 +1,57 @@ +#!/bin/bash + +# Begin $rc_base/init.d/pmacct + +# Starts and stops pmacct daemon +# Date: 2021-02-25 12:25:30 (Thu, 25 Feb 2021) + +# Locations +CONF="/etc/pmacct/pmacct.conf" +CONTENT="/var/pmacct" +# Pmacctd binary +PMACCT="/usr/sbin/pmacctd" +# Pmacctd start options '-f' for the configuration file +OPT="-f ${CONF}" +DESC="pmacct daemon" +# Pmacctd syslog facilitiy +#SYSLOG="local1" + + +. /etc/sysconfig/rc +. $rc_functions + +case "$1" in + start) + boot_mesg "Starting the ${DESC}... " + loadproc ${PMACCT} ${OPT} + sleep 2 + if P=$(pgrep -fl pmacctd); then + echo -e "\n${P}" + else + echo -e "\n${PMACCT} is not running... " + fi + ;; + + stop) + boot_mesg "Stopping the ${DESC}... " + killproc ${PMACCT} + ;; + + restart) + $0 stop + sleep 1 + $0 start + ;; + + status) + statusproc ${PMACCT} + ;; + + *) + echo "Usage: $0 {start|stop|restart|status}" + exit 1 + ;; +esac + +# End $rc_base/init.d/pmacct + diff --git a/config/pmacct/pmacct_memory_example.conf b/config/pmacct/pmacct_memory_example.conf new file mode 100644 index 000000000..0babdaaf7 --- /dev/null +++ b/config/pmacct/pmacct_memory_example.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +! + +syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0 + + +! +! "plugin1" plugin configuration +! +plugins: memory[plugin1] + +plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000 + +imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe + +aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip diff --git a/config/pmacct/pmacct_sqlite3_example.conf b/config/pmacct/pmacct_sqlite3_example.conf new file mode 100644 index 000000000..7fb996408 --- /dev/null +++ b/config/pmacct/pmacct_sqlite3_example.conf @@ -0,0 +1,25 @@ +! +! Pmacctd configuration file for IPFire environment +! + +syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0 + + +! +! "plugin2" plugin configuration +! +plugins: sqlite3[plugin2] + +sql_db[plugin2]: /var/spool/pmacct/pmacct_sqlitev1.db +sql_table_version[plugin2]: 1 +sql_history[plugin2]: 5m +sql_refresh_time[plugin2]: 300 +sql_history_roundoff[plugin2]: m +sql_dont_try_update: true + +aggregate[plugin2]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin2]: ip diff --git a/config/rootfiles/packages/pmacct b/config/rootfiles/packages/pmacct new file mode 100644 index 000000000..fed83ee67 --- /dev/null +++ b/config/rootfiles/packages/pmacct @@ -0,0 +1,111 @@ +etc/pmacct +etc/pmacct/pmacct.conf +etc/pmacct/pmacct_memory_example.conf +etc/pmacct/pmacct_sqlite3_example.conf +etc/rc.d/init.d/pmacct +usr/bin/pmacct +#usr/lib/pmacct +#usr/lib/pmacct/examples +#usr/lib/pmacct/examples/custom +#usr/lib/pmacct/examples/custom/libcustom.la +#usr/lib/pmacct/examples/lg +#usr/lib/pmacct/examples/lg/pmbgp +usr/sbin/nfacctd +usr/sbin/pmacctd +usr/sbin/pmbgpd +usr/sbin/pmbmpd +usr/sbin/pmtelemetryd +usr/sbin/sfacctd +usr/share/pmacct +#usr/share/pmacct/CONFIG-KEYS +#usr/share/pmacct/FAQS +#usr/share/pmacct/QUICKSTART +#usr/share/pmacct/UPGRADE +#usr/share/pmacct/docs +#usr/share/pmacct/docs/IPFIX +#usr/share/pmacct/docs/LOOKING_GLASS_FORMAT +#usr/share/pmacct/docs/MSGLOG_DUMP_FORMATS +#usr/share/pmacct/docs/SIGNALS +#usr/share/pmacct/docs/TRIGGER_VARS +#usr/share/pmacct/examples +usr/share/pmacct/examples/allow.lst.example +#usr/share/pmacct/examples/amqp +usr/share/pmacct/examples/amqp/amqp_receiver.py +#usr/share/pmacct/examples/avro +usr/share/pmacct/examples/avro/avro_file_decoder.py +usr/share/pmacct/examples/bgp_agent.map.example +usr/share/pmacct/examples/bgp_md5.lst.example +usr/share/pmacct/examples/bgp_xconnects.map.example +usr/share/pmacct/examples/flow_to_rd.map.example +#usr/share/pmacct/examples/kafka +usr/share/pmacct/examples/kafka/kafka_consumer.py +#usr/share/pmacct/examples/lg +usr/share/pmacct/examples/lg/pmbgp.py +usr/share/pmacct/examples/networks.lst.example +usr/share/pmacct/examples/nfacctd-sql.conf.example +usr/share/pmacct/examples/pcap_interfaces.map.example +usr/share/pmacct/examples/peers.map.example +usr/share/pmacct/examples/pmacctd-imt.conf.example +usr/share/pmacct/examples/pmacctd-multiple-plugins.conf.example +usr/share/pmacct/examples/pmacctd-sql.conf.example +usr/share/pmacct/examples/pmacctd-sqlite3.conf.example +usr/share/pmacct/examples/ports.lst.example +usr/share/pmacct/examples/pretag.map.example +usr/share/pmacct/examples/primitives.lst.example +usr/share/pmacct/examples/probe_netflow.conf.example +usr/share/pmacct/examples/probe_sflow.conf.example +usr/share/pmacct/examples/sampling.map.example +usr/share/pmacct/examples/tee_receivers.lst.example +#usr/share/pmacct/sql +#usr/share/pmacct/sql/README.64bit +#usr/share/pmacct/sql/README.GeoIP +#usr/share/pmacct/sql/README.IPv6 +#usr/share/pmacct/sql/README.cos +#usr/share/pmacct/sql/README.custom_primitives +#usr/share/pmacct/sql/README.etype +#usr/share/pmacct/sql/README.export_proto +#usr/share/pmacct/sql/README.iface +#usr/share/pmacct/sql/README.label +#usr/share/pmacct/sql/README.mask +#usr/share/pmacct/sql/README.mpls +#usr/share/pmacct/sql/README.mysql +#usr/share/pmacct/sql/README.nat +#usr/share/pmacct/sql/README.pgsql +#usr/share/pmacct/sql/README.sampling +usr/share/pmacct/sql/README.sqlite3 +#usr/share/pmacct/sql/README.tag2 +#usr/share/pmacct/sql/README.timestamp +#usr/share/pmacct/sql/README.tunnel +#usr/share/pmacct/sql/pmacct-create-db.pgsql +#usr/share/pmacct/sql/pmacct-create-db_bgp_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v2.mysql +#usr/share/pmacct/sql/pmacct-create-db_v3.mysql +#usr/share/pmacct/sql/pmacct-create-db_v4.mysql +#usr/share/pmacct/sql/pmacct-create-db_v5.mysql +#usr/share/pmacct/sql/pmacct-create-db_v6.mysql +#usr/share/pmacct/sql/pmacct-create-db_v7.mysql +#usr/share/pmacct/sql/pmacct-create-db_v8.mysql +#usr/share/pmacct/sql/pmacct-create-db_v9.mysql +#usr/share/pmacct/sql/pmacct-create-table_bgp_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_bgp_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v2.pgsql +usr/share/pmacct/sql/pmacct-create-table_v2.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v3.pgsql +usr/share/pmacct/sql/pmacct-create-table_v3.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v4.pgsql +usr/share/pmacct/sql/pmacct-create-table_v4.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v5.pgsql +usr/share/pmacct/sql/pmacct-create-table_v5.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v6.pgsql +usr/share/pmacct/sql/pmacct-create-table_v6.sqlite3 +usr/share/pmacct/sql/pmacct-create-table_v7.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v7_v8.pgsql +usr/share/pmacct/sql/pmacct-create-table_v8.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v9.pgsql +usr/share/pmacct/sql/pmacct-create-table_v9.sqlite3 +#usr/share/pmacct/sql/pmacct-grant-db.mysql +var/ipfire/backup/addons/includes/pmacct +var/spool/pmacct diff --git a/lfs/pmacct b/lfs/pmacct new file mode 100644 index 000000000..c6cc5270e --- /dev/null +++ b/lfs/pmacct @@ -0,0 +1,107 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2019 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +############################################################################### + +############################################################################### +# Definitions +############################################################################### + +include Config + +VER = 1.7.6 + +THISAPP = pmacct-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP) +PROG = pmacct +PAK_VER = 1 + +DEPS = libcdada + +############################################################################### +# Top-level Rules +############################################################################### + +objects = $(DL_FILE) + +$(DL_FILE) = $(DL_FROM)/$(DL_FILE) + +$(DL_FILE)_MD5 = b829cfdc394d5acd6d32fb23de181a6b + +install : $(TARGET) + +check : $(patsubst %,$(DIR_CHK)/%,$(objects)) + +download :$(patsubst %,$(DIR_DL)/%,$(objects)) + +md5 : $(subst %,%_MD5,$(objects)) + +dist: + @$(PAK) + +############################################################################### +# Downloading, checking, md5sum +############################################################################### + +$(patsubst %,$(DIR_CHK)/%,$(objects)) : + @$(CHECK) + +$(patsubst %,$(DIR_DL)/%,$(objects)) : + @$(LOAD) + +$(subst %,%_MD5,$(objects)) : + @$(MD5) + +############################################################################### +# Installation Details +############################################################################### + +$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) + @$(PREBUILD) + @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE) + cd $(DIR_APP) && ./autogen.sh + cd $(DIR_APP) && ./configure \ + --prefix=/usr \ + --sysconfdir=/etc/pmacct \ + --enable-sqlite3 \ + --enable-l2 \ + --enable-jansson + + # Add directory for plugins and pipes + -mkdir -v /var/spool/pmacct + + # Add configuration folder and copy conf files to it + -mkdir -v /etc/pmacct + cp -vf $(DIR_CONF)/pmacct/*.conf /etc/pmacct + + # Copy initscript to dir + cp -vf $(DIR_CONF)/pmacct/pmacct.init /etc/rc.d/init.d/pmacct + chmod 754 /etc/rc.d/init.d/pmacct + chown root:root /etc/rc.d/init.d/pmacct + + cd $(DIR_APP) && make $(MAKETUNING) + cd $(DIR_APP) && make install + + # Install backup definition + install -v -m 644 ${DIR_SRC}/config/backup/includes/pmacct \ + /var/ipfire/backup/addons/includes/pmacct + @rm -rf $(DIR_APP) + @$(POSTBUILD) diff --git a/make.sh b/make.sh index 43cccaca0..d42c9b1e2 100755 --- a/make.sh +++ b/make.sh @@ -1646,6 +1646,8 @@ buildipfire() { lfsmake2 ncdu lfsmake2 lshw lfsmake2 socat + lfsmake2 libcdada + lfsmake2 pmacct }
buildinstaller() { diff --git a/src/paks/pmacct/install.sh b/src/paks/pmacct/install.sh new file mode 100644 index 000000000..8178cc7bc --- /dev/null +++ b/src/paks/pmacct/install.sh @@ -0,0 +1,36 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh + +extract_files +restore_backup ${NAME} + +# Add symlinks for runlevels +ln -s ../init.d/${NAME} /etc/rc.d/rc0.d/K85${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc3.d/S50${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc6.d/K85${NAME} +start_service --background ${NAME} + + +# EOF diff --git a/src/paks/pmacct/uninstall.sh b/src/paks/pmacct/uninstall.sh new file mode 100644 index 000000000..51631e8c1 --- /dev/null +++ b/src/paks/pmacct/uninstall.sh @@ -0,0 +1,33 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh + +stop_service ${NAME} +make_backup ${NAME} +remove_files + +# Delete symlinks +rm -rfv /etc/rc.d/rc?.d/???${NAME} + +# EOF diff --git a/src/paks/pmacct/update.sh b/src/paks/pmacct/update.sh new file mode 100644 index 000000000..89c40d0d7 --- /dev/null +++ b/src/paks/pmacct/update.sh @@ -0,0 +1,26 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh +./uninstall.sh +./install.sh
Hello,
I will split this email into two parts:
The technical part:
* The initscript isn’t based on our template and uses sleep to check if starting a daemon has been successful. That should not be necessary and will potentially slow down the boot process.
* I am not very sure if the example configuration files are helpful. Maybe someone can clear this up a little bit for me what is the rationale is behind them.
* There are loads of daemons being shipped and only one is being used.
* How much load does this generate? How large is the captured data?
Then there is the other part:
On 8 Mar 2021, at 11:57, Adolf Belka (ipfire) adolf.belka@ipfire.org wrote:
From: Adolf Belka adolf.belka@ipfire.org
- What is it? pmacct is a monitoring tool for network management tasks. Data collected can be used for analysis and troubleshooting purposes to maintain the health of the network. pmacct can collect, replicate and export network information. It can cache in memory tables, store persistently to SQLite3 and output to flat-files like CSV, formatted, and JSON.
Okay.
- Why is it needed? To monitor data usage (IP-based or MAC-based data accounting) down to the client level. Net-Traffic will monitor traffic for the entire RED, GREEN, etc. networks, but it cannot pinpoint which client is using lots of data. Connections will take a snapshot but not show day by day sums. pmacct can help admins keep tabs on users that use too much data.
I agree with the goal of it. But how does this add-on achieve that?
The data is being written to a (usually binary) file which cannot be visualised in the web UI. Does this not want a CGI script that allows people to see what is happening?
Sending manual SQL queries probably isn’t much fun :)
- What are the use cases? An ISP may implement data caps and if the limit is over-run then you have to pay for every additional xxGB of data used. Typical charges can be around $10 per 50GB. With pmacct you can identify the high users and take action, hopefully before the limit is breached.
- This is being introduced as a command line only tool. However, at a later date, if it is useful to enough additional users a WUI page could be developed as discussed in the development mailing list https://lists.ipfire.org/pipermail/development/2021-January/009174.html
I have been thinking about this for a while now and would say that I can approve merging this as a CLI tool. But it would not really enhance IPFire’s functionality because it is only usable for people who spend a lot of time writing their own scripts or SQL queries to get information that can be used to make decisions.
Adding another alternative to vnstat will result in us having to maintain two things that roughly do the same - or where one’s features are a subset of the other one.
I would like to avoid merging this and then see it abandoned because it is too complicated to use. We already have a lot of virtually unmaintained stuff around and I would really like to drop a lot of it, although I know that there are a few people actually using it.
Thoughts?
Best, -Michael
Co-authored-by: Jon Murphy Co-authored-by: Erik Kapfer ummeegge@ipfire.org
Signed-off-by: Adolf Belka adolf.belka@ipfire.org
config/backup/includes/pmacct | 2 + config/pmacct/pmacct.conf | 24 +++++ config/pmacct/pmacct.init | 57 +++++++++++ config/pmacct/pmacct_memory_example.conf | 24 +++++ config/pmacct/pmacct_sqlite3_example.conf | 25 +++++ config/rootfiles/packages/pmacct | 111 ++++++++++++++++++++++ lfs/pmacct | 107 +++++++++++++++++++++ make.sh | 2 + src/paks/pmacct/install.sh | 36 +++++++ src/paks/pmacct/uninstall.sh | 33 +++++++ src/paks/pmacct/update.sh | 26 +++++ 11 files changed, 447 insertions(+) create mode 100644 config/backup/includes/pmacct create mode 100644 config/pmacct/pmacct.conf create mode 100644 config/pmacct/pmacct.init create mode 100644 config/pmacct/pmacct_memory_example.conf create mode 100644 config/pmacct/pmacct_sqlite3_example.conf create mode 100644 config/rootfiles/packages/pmacct create mode 100644 lfs/pmacct create mode 100644 src/paks/pmacct/install.sh create mode 100644 src/paks/pmacct/uninstall.sh create mode 100644 src/paks/pmacct/update.sh
diff --git a/config/backup/includes/pmacct b/config/backup/includes/pmacct new file mode 100644 index 000000000..4359067ad --- /dev/null +++ b/config/backup/includes/pmacct @@ -0,0 +1,2 @@ +/etc/pmacct/ +/var/spool/pmacct/ diff --git a/config/pmacct/pmacct.conf b/config/pmacct/pmacct.conf new file mode 100644 index 000000000..3c1d47efa --- /dev/null +++ b/config/pmacct/pmacct.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip \ No newline at end of file diff --git a/config/pmacct/pmacct.init b/config/pmacct/pmacct.init new file mode 100644 index 000000000..31f9ce49b --- /dev/null +++ b/config/pmacct/pmacct.init @@ -0,0 +1,57 @@ +#!/bin/bash
+# Begin $rc_base/init.d/pmacct
+# Starts and stops pmacct daemon +# Date: 2021-02-25 12:25:30 (Thu, 25 Feb 2021)
+# Locations +CONF="/etc/pmacct/pmacct.conf" +CONTENT="/var/pmacct" +# Pmacctd binary +PMACCT="/usr/sbin/pmacctd" +# Pmacctd start options '-f' for the configuration file +OPT="-f ${CONF}" +DESC="pmacct daemon" +# Pmacctd syslog facilitiy +#SYSLOG="local1"
+. /etc/sysconfig/rc +. $rc_functions
+case "$1" in
- start)
boot_mesg "Starting the ${DESC}... "
loadproc ${PMACCT} ${OPT}
sleep 2
if P=$(pgrep -fl pmacctd); then
echo -e "\n${P}"
else
echo -e "\n${PMACCT} is not running... "
fi
;;
- stop)
boot_mesg "Stopping the ${DESC}... "
killproc ${PMACCT}
;;
- restart)
$0 stop
sleep 1
$0 start
;;
- status)
statusproc ${PMACCT}
;;
- *)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
+esac
+# End $rc_base/init.d/pmacct
diff --git a/config/pmacct/pmacct_memory_example.conf b/config/pmacct/pmacct_memory_example.conf new file mode 100644 index 000000000..0babdaaf7 --- /dev/null +++ b/config/pmacct/pmacct_memory_example.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip diff --git a/config/pmacct/pmacct_sqlite3_example.conf b/config/pmacct/pmacct_sqlite3_example.conf new file mode 100644 index 000000000..7fb996408 --- /dev/null +++ b/config/pmacct/pmacct_sqlite3_example.conf @@ -0,0 +1,25 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin2" plugin configuration +! +plugins: sqlite3[plugin2]
+sql_db[plugin2]: /var/spool/pmacct/pmacct_sqlitev1.db +sql_table_version[plugin2]: 1 +sql_history[plugin2]: 5m +sql_refresh_time[plugin2]: 300 +sql_history_roundoff[plugin2]: m +sql_dont_try_update: true
+aggregate[plugin2]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin2]: ip diff --git a/config/rootfiles/packages/pmacct b/config/rootfiles/packages/pmacct new file mode 100644 index 000000000..fed83ee67 --- /dev/null +++ b/config/rootfiles/packages/pmacct @@ -0,0 +1,111 @@ +etc/pmacct +etc/pmacct/pmacct.conf +etc/pmacct/pmacct_memory_example.conf +etc/pmacct/pmacct_sqlite3_example.conf +etc/rc.d/init.d/pmacct +usr/bin/pmacct +#usr/lib/pmacct +#usr/lib/pmacct/examples +#usr/lib/pmacct/examples/custom +#usr/lib/pmacct/examples/custom/libcustom.la +#usr/lib/pmacct/examples/lg +#usr/lib/pmacct/examples/lg/pmbgp +usr/sbin/nfacctd +usr/sbin/pmacctd +usr/sbin/pmbgpd +usr/sbin/pmbmpd +usr/sbin/pmtelemetryd +usr/sbin/sfacctd +usr/share/pmacct +#usr/share/pmacct/CONFIG-KEYS +#usr/share/pmacct/FAQS +#usr/share/pmacct/QUICKSTART +#usr/share/pmacct/UPGRADE +#usr/share/pmacct/docs +#usr/share/pmacct/docs/IPFIX +#usr/share/pmacct/docs/LOOKING_GLASS_FORMAT +#usr/share/pmacct/docs/MSGLOG_DUMP_FORMATS +#usr/share/pmacct/docs/SIGNALS +#usr/share/pmacct/docs/TRIGGER_VARS +#usr/share/pmacct/examples +usr/share/pmacct/examples/allow.lst.example +#usr/share/pmacct/examples/amqp +usr/share/pmacct/examples/amqp/amqp_receiver.py +#usr/share/pmacct/examples/avro +usr/share/pmacct/examples/avro/avro_file_decoder.py +usr/share/pmacct/examples/bgp_agent.map.example +usr/share/pmacct/examples/bgp_md5.lst.example +usr/share/pmacct/examples/bgp_xconnects.map.example +usr/share/pmacct/examples/flow_to_rd.map.example +#usr/share/pmacct/examples/kafka +usr/share/pmacct/examples/kafka/kafka_consumer.py +#usr/share/pmacct/examples/lg +usr/share/pmacct/examples/lg/pmbgp.py +usr/share/pmacct/examples/networks.lst.example +usr/share/pmacct/examples/nfacctd-sql.conf.example +usr/share/pmacct/examples/pcap_interfaces.map.example +usr/share/pmacct/examples/peers.map.example +usr/share/pmacct/examples/pmacctd-imt.conf.example +usr/share/pmacct/examples/pmacctd-multiple-plugins.conf.example +usr/share/pmacct/examples/pmacctd-sql.conf.example +usr/share/pmacct/examples/pmacctd-sqlite3.conf.example +usr/share/pmacct/examples/ports.lst.example +usr/share/pmacct/examples/pretag.map.example +usr/share/pmacct/examples/primitives.lst.example +usr/share/pmacct/examples/probe_netflow.conf.example +usr/share/pmacct/examples/probe_sflow.conf.example +usr/share/pmacct/examples/sampling.map.example +usr/share/pmacct/examples/tee_receivers.lst.example +#usr/share/pmacct/sql +#usr/share/pmacct/sql/README.64bit +#usr/share/pmacct/sql/README.GeoIP +#usr/share/pmacct/sql/README.IPv6 +#usr/share/pmacct/sql/README.cos +#usr/share/pmacct/sql/README.custom_primitives +#usr/share/pmacct/sql/README.etype +#usr/share/pmacct/sql/README.export_proto +#usr/share/pmacct/sql/README.iface +#usr/share/pmacct/sql/README.label +#usr/share/pmacct/sql/README.mask +#usr/share/pmacct/sql/README.mpls +#usr/share/pmacct/sql/README.mysql +#usr/share/pmacct/sql/README.nat +#usr/share/pmacct/sql/README.pgsql +#usr/share/pmacct/sql/README.sampling +usr/share/pmacct/sql/README.sqlite3 +#usr/share/pmacct/sql/README.tag2 +#usr/share/pmacct/sql/README.timestamp +#usr/share/pmacct/sql/README.tunnel +#usr/share/pmacct/sql/pmacct-create-db.pgsql +#usr/share/pmacct/sql/pmacct-create-db_bgp_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v2.mysql +#usr/share/pmacct/sql/pmacct-create-db_v3.mysql +#usr/share/pmacct/sql/pmacct-create-db_v4.mysql +#usr/share/pmacct/sql/pmacct-create-db_v5.mysql +#usr/share/pmacct/sql/pmacct-create-db_v6.mysql +#usr/share/pmacct/sql/pmacct-create-db_v7.mysql +#usr/share/pmacct/sql/pmacct-create-db_v8.mysql +#usr/share/pmacct/sql/pmacct-create-db_v9.mysql +#usr/share/pmacct/sql/pmacct-create-table_bgp_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_bgp_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v2.pgsql +usr/share/pmacct/sql/pmacct-create-table_v2.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v3.pgsql +usr/share/pmacct/sql/pmacct-create-table_v3.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v4.pgsql +usr/share/pmacct/sql/pmacct-create-table_v4.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v5.pgsql +usr/share/pmacct/sql/pmacct-create-table_v5.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v6.pgsql +usr/share/pmacct/sql/pmacct-create-table_v6.sqlite3 +usr/share/pmacct/sql/pmacct-create-table_v7.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v7_v8.pgsql +usr/share/pmacct/sql/pmacct-create-table_v8.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v9.pgsql +usr/share/pmacct/sql/pmacct-create-table_v9.sqlite3 +#usr/share/pmacct/sql/pmacct-grant-db.mysql +var/ipfire/backup/addons/includes/pmacct +var/spool/pmacct diff --git a/lfs/pmacct b/lfs/pmacct new file mode 100644 index 000000000..c6cc5270e --- /dev/null +++ b/lfs/pmacct @@ -0,0 +1,107 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2019 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################
+############################################################################### +# Definitions +###############################################################################
+include Config
+VER = 1.7.6
+THISAPP = pmacct-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP) +PROG = pmacct +PAK_VER = 1
+DEPS = libcdada
+############################################################################### +# Top-level Rules +###############################################################################
+objects = $(DL_FILE)
+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
+$(DL_FILE)_MD5 = b829cfdc394d5acd6d32fb23de181a6b
+install : $(TARGET)
+check : $(patsubst %,$(DIR_CHK)/%,$(objects))
+download :$(patsubst %,$(DIR_DL)/%,$(objects))
+md5 : $(subst %,%_MD5,$(objects))
+dist:
- @$(PAK)
+############################################################################### +# Downloading, checking, md5sum +###############################################################################
+$(patsubst %,$(DIR_CHK)/%,$(objects)) :
- @$(CHECK)
+$(patsubst %,$(DIR_DL)/%,$(objects)) :
- @$(LOAD)
+$(subst %,%_MD5,$(objects)) :
- @$(MD5)
+############################################################################### +# Installation Details +###############################################################################
+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
- @$(PREBUILD)
- @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE)
- cd $(DIR_APP) && ./autogen.sh
- cd $(DIR_APP) && ./configure \
--prefix=/usr \
--sysconfdir=/etc/pmacct \
--enable-sqlite3 \
--enable-l2 \
--enable-jansson
- # Add directory for plugins and pipes
- -mkdir -v /var/spool/pmacct
- # Add configuration folder and copy conf files to it
- -mkdir -v /etc/pmacct
- cp -vf $(DIR_CONF)/pmacct/*.conf /etc/pmacct
- # Copy initscript to dir
- cp -vf $(DIR_CONF)/pmacct/pmacct.init /etc/rc.d/init.d/pmacct
- chmod 754 /etc/rc.d/init.d/pmacct
- chown root:root /etc/rc.d/init.d/pmacct
- cd $(DIR_APP) && make $(MAKETUNING)
- cd $(DIR_APP) && make install
- # Install backup definition
- install -v -m 644 ${DIR_SRC}/config/backup/includes/pmacct \
/var/ipfire/backup/addons/includes/pmacct
- @rm -rf $(DIR_APP)
- @$(POSTBUILD)
diff --git a/make.sh b/make.sh index 43cccaca0..d42c9b1e2 100755 --- a/make.sh +++ b/make.sh @@ -1646,6 +1646,8 @@ buildipfire() { lfsmake2 ncdu lfsmake2 lshw lfsmake2 socat
- lfsmake2 libcdada
- lfsmake2 pmacct
}
buildinstaller() { diff --git a/src/paks/pmacct/install.sh b/src/paks/pmacct/install.sh new file mode 100644 index 000000000..8178cc7bc --- /dev/null +++ b/src/paks/pmacct/install.sh @@ -0,0 +1,36 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh
+extract_files +restore_backup ${NAME}
+# Add symlinks for runlevels +ln -s ../init.d/${NAME} /etc/rc.d/rc0.d/K85${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc3.d/S50${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc6.d/K85${NAME} +start_service --background ${NAME}
+# EOF diff --git a/src/paks/pmacct/uninstall.sh b/src/paks/pmacct/uninstall.sh new file mode 100644 index 000000000..51631e8c1 --- /dev/null +++ b/src/paks/pmacct/uninstall.sh @@ -0,0 +1,33 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh
+stop_service ${NAME} +make_backup ${NAME} +remove_files
+# Delete symlinks +rm -rfv /etc/rc.d/rc?.d/???${NAME}
+# EOF diff --git a/src/paks/pmacct/update.sh b/src/paks/pmacct/update.sh new file mode 100644 index 000000000..89c40d0d7 --- /dev/null +++ b/src/paks/pmacct/update.sh @@ -0,0 +1,26 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh +./uninstall.sh
+./install.sh
2.30.1
Hi Michael,
On 18/03/2021 14:07, Michael Tremer wrote:
Hello,
I will split this email into two parts:
The technical part:
- The initscript isn’t based on our template and uses sleep to check if starting a daemon has been successful. That should not be necessary and will potentially slow down the boot process.
I will have another look at the initscript and see if it can use the standard IPFire template.
- I am not very sure if the example configuration files are helpful. Maybe someone can clear this up a little bit for me what is the rationale is behind them.
The idea was that the two simplest config example scripts were provided that give a user the ability to get data into memory or into a sqlite file. The actual conf file provided is a copy of the memory version. These give a basic starting point that people can then add additional info to be recorded etc.
- There are loads of daemons being shipped and only one is being used.
I think Jon or Erik need to comment on this question.
- How much load does this generate? How large is the captured data?
Jon needs to provide this feedback.
Then there is the other part:
On 8 Mar 2021, at 11:57, Adolf Belka (ipfire) adolf.belka@ipfire.org wrote:
From: Adolf Belka adolf.belka@ipfire.org
- What is it? pmacct is a monitoring tool for network management tasks. Data collected can be used for analysis and troubleshooting purposes to maintain the health of the network. pmacct can collect, replicate and export network information. It can cache in memory tables, store persistently to SQLite3 and output to flat-files like CSV, formatted, and JSON.
Okay.
- Why is it needed? To monitor data usage (IP-based or MAC-based data accounting) down to the client level. Net-Traffic will monitor traffic for the entire RED, GREEN, etc. networks, but it cannot pinpoint which client is using lots of data. Connections will take a snapshot but not show day by day sums. pmacct can help admins keep tabs on users that use too much data.
I agree with the goal of it. But how does this add-on achieve that?
Jon should comment on this as he had this use case and can give feedback how the data has helped him in his investigations.
The data is being written to a (usually binary) file which cannot be visualised in the web UI. Does this not want a CGI script that allows people to see what is happening?
We all thought that this would ultimately need a CGI script to set up and visualise the data. We believed it was worth getting this running in actual use as a CLI tool to start with and then get more feedback on the best items that would need to be captured in the CGI script. The tool has a lot of capabilities and I think it will not be possible to do everything that it can do in a WUI page. So some form of file or db offload will need to be available anyway but the items identified as being commonly required would be the ones to be developed into a CGI WUI page.
Sending manual SQL queries probably isn’t much fun :)
- What are the use cases? An ISP may implement data caps and if the limit is over-run then you have to pay for every additional xxGB of data used. Typical charges can be around $10 per 50GB. With pmacct you can identify the high users and take action, hopefully before the limit is breached.
- This is being introduced as a command line only tool. However, at a later date, if it is useful to enough additional users a WUI page could be developed as discussed in the development mailing list https://lists.ipfire.org/pipermail/development/2021-January/009174.html
I have been thinking about this for a while now and would say that I can approve merging this as a CLI tool. But it would not really enhance IPFire’s functionality because it is only usable for people who spend a lot of time writing their own scripts or SQL queries to get information that can be used to make decisions.
We recognise that and had discussed the CGI WUI page as mentioned above.
Adding another alternative to vnstat will result in us having to maintain two things that roughly do the same - or where one’s features are a subset of the other one.
I think Jon or Erik need to comment on any comparison with vnstat.
I would like to avoid merging this and then see it abandoned because it is too complicated to use. We already have a lot of virtually unmaintained stuff around and I would really like to drop a lot of it, although I know that there are a few people actually using it.
Jon and Erik should comment on the complication of use or not. Jon especially has done quite a bit of work with it, installing the .ipfire package versions that I have created. I intend to also have a play around with it but for me it is more some playful fun as I just have a home network with only myself as the user and admin.
Thoughts?
Best, -Michael
Co-authored-by: Jon Murphy Co-authored-by: Erik Kapfer ummeegge@ipfire.org
Signed-off-by: Adolf Belka adolf.belka@ipfire.org
config/backup/includes/pmacct | 2 + config/pmacct/pmacct.conf | 24 +++++ config/pmacct/pmacct.init | 57 +++++++++++ config/pmacct/pmacct_memory_example.conf | 24 +++++ config/pmacct/pmacct_sqlite3_example.conf | 25 +++++ config/rootfiles/packages/pmacct | 111 ++++++++++++++++++++++ lfs/pmacct | 107 +++++++++++++++++++++ make.sh | 2 + src/paks/pmacct/install.sh | 36 +++++++ src/paks/pmacct/uninstall.sh | 33 +++++++ src/paks/pmacct/update.sh | 26 +++++ 11 files changed, 447 insertions(+) create mode 100644 config/backup/includes/pmacct create mode 100644 config/pmacct/pmacct.conf create mode 100644 config/pmacct/pmacct.init create mode 100644 config/pmacct/pmacct_memory_example.conf create mode 100644 config/pmacct/pmacct_sqlite3_example.conf create mode 100644 config/rootfiles/packages/pmacct create mode 100644 lfs/pmacct create mode 100644 src/paks/pmacct/install.sh create mode 100644 src/paks/pmacct/uninstall.sh create mode 100644 src/paks/pmacct/update.sh
diff --git a/config/backup/includes/pmacct b/config/backup/includes/pmacct new file mode 100644 index 000000000..4359067ad --- /dev/null +++ b/config/backup/includes/pmacct @@ -0,0 +1,2 @@ +/etc/pmacct/ +/var/spool/pmacct/ diff --git a/config/pmacct/pmacct.conf b/config/pmacct/pmacct.conf new file mode 100644 index 000000000..3c1d47efa --- /dev/null +++ b/config/pmacct/pmacct.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip \ No newline at end of file diff --git a/config/pmacct/pmacct.init b/config/pmacct/pmacct.init new file mode 100644 index 000000000..31f9ce49b --- /dev/null +++ b/config/pmacct/pmacct.init @@ -0,0 +1,57 @@ +#!/bin/bash
+# Begin $rc_base/init.d/pmacct
+# Starts and stops pmacct daemon +# Date: 2021-02-25 12:25:30 (Thu, 25 Feb 2021)
+# Locations +CONF="/etc/pmacct/pmacct.conf" +CONTENT="/var/pmacct" +# Pmacctd binary +PMACCT="/usr/sbin/pmacctd" +# Pmacctd start options '-f' for the configuration file +OPT="-f ${CONF}" +DESC="pmacct daemon" +# Pmacctd syslog facilitiy +#SYSLOG="local1"
+. /etc/sysconfig/rc +. $rc_functions
+case "$1" in
- start)
boot_mesg "Starting the ${DESC}... "
loadproc ${PMACCT} ${OPT}
sleep 2
if P=$(pgrep -fl pmacctd); then
echo -e "\n${P}"
else
echo -e "\n${PMACCT} is not running... "
fi
;;
- stop)
boot_mesg "Stopping the ${DESC}... "
killproc ${PMACCT}
;;
- restart)
$0 stop
sleep 1
$0 start
;;
- status)
statusproc ${PMACCT}
;;
- *)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
+esac
+# End $rc_base/init.d/pmacct
diff --git a/config/pmacct/pmacct_memory_example.conf b/config/pmacct/pmacct_memory_example.conf new file mode 100644 index 000000000..0babdaaf7 --- /dev/null +++ b/config/pmacct/pmacct_memory_example.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip diff --git a/config/pmacct/pmacct_sqlite3_example.conf b/config/pmacct/pmacct_sqlite3_example.conf new file mode 100644 index 000000000..7fb996408 --- /dev/null +++ b/config/pmacct/pmacct_sqlite3_example.conf @@ -0,0 +1,25 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin2" plugin configuration +! +plugins: sqlite3[plugin2]
+sql_db[plugin2]: /var/spool/pmacct/pmacct_sqlitev1.db +sql_table_version[plugin2]: 1 +sql_history[plugin2]: 5m +sql_refresh_time[plugin2]: 300 +sql_history_roundoff[plugin2]: m +sql_dont_try_update: true
+aggregate[plugin2]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin2]: ip diff --git a/config/rootfiles/packages/pmacct b/config/rootfiles/packages/pmacct new file mode 100644 index 000000000..fed83ee67 --- /dev/null +++ b/config/rootfiles/packages/pmacct @@ -0,0 +1,111 @@ +etc/pmacct +etc/pmacct/pmacct.conf +etc/pmacct/pmacct_memory_example.conf +etc/pmacct/pmacct_sqlite3_example.conf +etc/rc.d/init.d/pmacct +usr/bin/pmacct +#usr/lib/pmacct +#usr/lib/pmacct/examples +#usr/lib/pmacct/examples/custom +#usr/lib/pmacct/examples/custom/libcustom.la +#usr/lib/pmacct/examples/lg +#usr/lib/pmacct/examples/lg/pmbgp +usr/sbin/nfacctd +usr/sbin/pmacctd +usr/sbin/pmbgpd +usr/sbin/pmbmpd +usr/sbin/pmtelemetryd +usr/sbin/sfacctd +usr/share/pmacct +#usr/share/pmacct/CONFIG-KEYS +#usr/share/pmacct/FAQS +#usr/share/pmacct/QUICKSTART +#usr/share/pmacct/UPGRADE +#usr/share/pmacct/docs +#usr/share/pmacct/docs/IPFIX +#usr/share/pmacct/docs/LOOKING_GLASS_FORMAT +#usr/share/pmacct/docs/MSGLOG_DUMP_FORMATS +#usr/share/pmacct/docs/SIGNALS +#usr/share/pmacct/docs/TRIGGER_VARS +#usr/share/pmacct/examples +usr/share/pmacct/examples/allow.lst.example +#usr/share/pmacct/examples/amqp +usr/share/pmacct/examples/amqp/amqp_receiver.py +#usr/share/pmacct/examples/avro +usr/share/pmacct/examples/avro/avro_file_decoder.py +usr/share/pmacct/examples/bgp_agent.map.example +usr/share/pmacct/examples/bgp_md5.lst.example +usr/share/pmacct/examples/bgp_xconnects.map.example +usr/share/pmacct/examples/flow_to_rd.map.example +#usr/share/pmacct/examples/kafka +usr/share/pmacct/examples/kafka/kafka_consumer.py +#usr/share/pmacct/examples/lg +usr/share/pmacct/examples/lg/pmbgp.py +usr/share/pmacct/examples/networks.lst.example +usr/share/pmacct/examples/nfacctd-sql.conf.example +usr/share/pmacct/examples/pcap_interfaces.map.example +usr/share/pmacct/examples/peers.map.example +usr/share/pmacct/examples/pmacctd-imt.conf.example +usr/share/pmacct/examples/pmacctd-multiple-plugins.conf.example +usr/share/pmacct/examples/pmacctd-sql.conf.example +usr/share/pmacct/examples/pmacctd-sqlite3.conf.example +usr/share/pmacct/examples/ports.lst.example +usr/share/pmacct/examples/pretag.map.example +usr/share/pmacct/examples/primitives.lst.example +usr/share/pmacct/examples/probe_netflow.conf.example +usr/share/pmacct/examples/probe_sflow.conf.example +usr/share/pmacct/examples/sampling.map.example +usr/share/pmacct/examples/tee_receivers.lst.example +#usr/share/pmacct/sql +#usr/share/pmacct/sql/README.64bit +#usr/share/pmacct/sql/README.GeoIP +#usr/share/pmacct/sql/README.IPv6 +#usr/share/pmacct/sql/README.cos +#usr/share/pmacct/sql/README.custom_primitives +#usr/share/pmacct/sql/README.etype +#usr/share/pmacct/sql/README.export_proto +#usr/share/pmacct/sql/README.iface +#usr/share/pmacct/sql/README.label +#usr/share/pmacct/sql/README.mask +#usr/share/pmacct/sql/README.mpls +#usr/share/pmacct/sql/README.mysql +#usr/share/pmacct/sql/README.nat +#usr/share/pmacct/sql/README.pgsql +#usr/share/pmacct/sql/README.sampling +usr/share/pmacct/sql/README.sqlite3 +#usr/share/pmacct/sql/README.tag2 +#usr/share/pmacct/sql/README.timestamp +#usr/share/pmacct/sql/README.tunnel +#usr/share/pmacct/sql/pmacct-create-db.pgsql +#usr/share/pmacct/sql/pmacct-create-db_bgp_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v2.mysql +#usr/share/pmacct/sql/pmacct-create-db_v3.mysql +#usr/share/pmacct/sql/pmacct-create-db_v4.mysql +#usr/share/pmacct/sql/pmacct-create-db_v5.mysql +#usr/share/pmacct/sql/pmacct-create-db_v6.mysql +#usr/share/pmacct/sql/pmacct-create-db_v7.mysql +#usr/share/pmacct/sql/pmacct-create-db_v8.mysql +#usr/share/pmacct/sql/pmacct-create-db_v9.mysql +#usr/share/pmacct/sql/pmacct-create-table_bgp_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_bgp_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v2.pgsql +usr/share/pmacct/sql/pmacct-create-table_v2.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v3.pgsql +usr/share/pmacct/sql/pmacct-create-table_v3.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v4.pgsql +usr/share/pmacct/sql/pmacct-create-table_v4.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v5.pgsql +usr/share/pmacct/sql/pmacct-create-table_v5.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v6.pgsql +usr/share/pmacct/sql/pmacct-create-table_v6.sqlite3 +usr/share/pmacct/sql/pmacct-create-table_v7.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v7_v8.pgsql +usr/share/pmacct/sql/pmacct-create-table_v8.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v9.pgsql +usr/share/pmacct/sql/pmacct-create-table_v9.sqlite3 +#usr/share/pmacct/sql/pmacct-grant-db.mysql +var/ipfire/backup/addons/includes/pmacct +var/spool/pmacct diff --git a/lfs/pmacct b/lfs/pmacct new file mode 100644 index 000000000..c6cc5270e --- /dev/null +++ b/lfs/pmacct @@ -0,0 +1,107 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2019 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################
+############################################################################### +# Definitions +###############################################################################
+include Config
+VER = 1.7.6
+THISAPP = pmacct-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP) +PROG = pmacct +PAK_VER = 1
+DEPS = libcdada
+############################################################################### +# Top-level Rules +###############################################################################
+objects = $(DL_FILE)
+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
+$(DL_FILE)_MD5 = b829cfdc394d5acd6d32fb23de181a6b
+install : $(TARGET)
+check : $(patsubst %,$(DIR_CHK)/%,$(objects))
+download :$(patsubst %,$(DIR_DL)/%,$(objects))
+md5 : $(subst %,%_MD5,$(objects))
+dist:
- @$(PAK)
+############################################################################### +# Downloading, checking, md5sum +###############################################################################
+$(patsubst %,$(DIR_CHK)/%,$(objects)) :
- @$(CHECK)
+$(patsubst %,$(DIR_DL)/%,$(objects)) :
- @$(LOAD)
+$(subst %,%_MD5,$(objects)) :
- @$(MD5)
+############################################################################### +# Installation Details +###############################################################################
+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
- @$(PREBUILD)
- @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE)
- cd $(DIR_APP) && ./autogen.sh
- cd $(DIR_APP) && ./configure \
--prefix=/usr \
--sysconfdir=/etc/pmacct \
--enable-sqlite3 \
--enable-l2 \
--enable-jansson
- # Add directory for plugins and pipes
- -mkdir -v /var/spool/pmacct
- # Add configuration folder and copy conf files to it
- -mkdir -v /etc/pmacct
- cp -vf $(DIR_CONF)/pmacct/*.conf /etc/pmacct
- # Copy initscript to dir
- cp -vf $(DIR_CONF)/pmacct/pmacct.init /etc/rc.d/init.d/pmacct
- chmod 754 /etc/rc.d/init.d/pmacct
- chown root:root /etc/rc.d/init.d/pmacct
- cd $(DIR_APP) && make $(MAKETUNING)
- cd $(DIR_APP) && make install
- # Install backup definition
- install -v -m 644 ${DIR_SRC}/config/backup/includes/pmacct \
/var/ipfire/backup/addons/includes/pmacct
- @rm -rf $(DIR_APP)
- @$(POSTBUILD)
diff --git a/make.sh b/make.sh index 43cccaca0..d42c9b1e2 100755 --- a/make.sh +++ b/make.sh @@ -1646,6 +1646,8 @@ buildipfire() { lfsmake2 ncdu lfsmake2 lshw lfsmake2 socat
- lfsmake2 libcdada
- lfsmake2 pmacct
}
buildinstaller() { diff --git a/src/paks/pmacct/install.sh b/src/paks/pmacct/install.sh new file mode 100644 index 000000000..8178cc7bc --- /dev/null +++ b/src/paks/pmacct/install.sh @@ -0,0 +1,36 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh
+extract_files +restore_backup ${NAME}
+# Add symlinks for runlevels +ln -s ../init.d/${NAME} /etc/rc.d/rc0.d/K85${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc3.d/S50${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc6.d/K85${NAME} +start_service --background ${NAME}
+# EOF diff --git a/src/paks/pmacct/uninstall.sh b/src/paks/pmacct/uninstall.sh new file mode 100644 index 000000000..51631e8c1 --- /dev/null +++ b/src/paks/pmacct/uninstall.sh @@ -0,0 +1,33 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh
+stop_service ${NAME} +make_backup ${NAME} +remove_files
+# Delete symlinks +rm -rfv /etc/rc.d/rc?.d/???${NAME}
+# EOF diff --git a/src/paks/pmacct/update.sh b/src/paks/pmacct/update.sh new file mode 100644 index 000000000..89c40d0d7 --- /dev/null +++ b/src/paks/pmacct/update.sh @@ -0,0 +1,26 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh +./uninstall.sh
+./install.sh
2.30.1
Hey,
On 18 Mar 2021, at 13:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 18/03/2021 14:07, Michael Tremer wrote:
Hello, I will split this email into two parts: The technical part:
- The initscript isn’t based on our template and uses sleep to check if starting a daemon has been successful. That should not be necessary and will potentially slow down the boot process.
I will have another look at the initscript and see if it can use the standard IPFire template.
Yes, it is not too different from your version:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/initscripts/system/tem...
There isn’t much documentation around it but there are loads of other scripts that will be helpful to figure out any convention or boilerplate code to copy.
- I am not very sure if the example configuration files are helpful. Maybe someone can clear this up a little bit for me what is the rationale is behind them.
The idea was that the two simplest config example scripts were provided that give a user the ability to get data into memory or into a sqlite file. The actual conf file provided is a copy of the memory version. These give a basic starting point that people can then add additional info to be recorded etc.
Does anybody want to use anything else than SQLite in the context of IPFire?
- There are loads of daemons being shipped and only one is being used.
I think Jon or Erik need to comment on this question.
- How much load does this generate? How large is the captured data?
Jon needs to provide this feedback.
Then there is the other part:
On 8 Mar 2021, at 11:57, Adolf Belka (ipfire) adolf.belka@ipfire.org wrote:
From: Adolf Belka adolf.belka@ipfire.org
- What is it? pmacct is a monitoring tool for network management tasks. Data collected can be used for analysis and troubleshooting purposes to maintain the health of the network. pmacct can collect, replicate and export network information. It can cache in memory tables, store persistently to SQLite3 and output to flat-files like CSV, formatted, and JSON.
Okay.
- Why is it needed? To monitor data usage (IP-based or MAC-based data accounting) down to the client level. Net-Traffic will monitor traffic for the entire RED, GREEN, etc. networks, but it cannot pinpoint which client is using lots of data. Connections will take a snapshot but not show day by day sums. pmacct can help admins keep tabs on users that use too much data.
I agree with the goal of it. But how does this add-on achieve that?
Jon should comment on this as he had this use case and can give feedback how the data has helped him in his investigations.
The data is being written to a (usually binary) file which cannot be visualised in the web UI. Does this not want a CGI script that allows people to see what is happening?
We all thought that this would ultimately need a CGI script to set up and visualise the data. We believed it was worth getting this running in actual use as a CLI tool to start with and then get more feedback on the best items that would need to be captured in the CGI script. The tool has a lot of capabilities and I think it will not be possible to do everything that it can do in a WUI page. So some form of file or db offload will need to be available anyway but the items identified as being commonly required would be the ones to be developed into a CGI WUI page.
Sending manual SQL queries probably isn’t much fun :)
- What are the use cases? An ISP may implement data caps and if the limit is over-run then you have to pay for every additional xxGB of data used. Typical charges can be around $10 per 50GB. With pmacct you can identify the high users and take action, hopefully before the limit is breached.
- This is being introduced as a command line only tool. However, at a later date, if it is useful to enough additional users a WUI page could be developed as discussed in the development mailing list https://lists.ipfire.org/pipermail/development/2021-January/009174.html
I have been thinking about this for a while now and would say that I can approve merging this as a CLI tool. But it would not really enhance IPFire’s functionality because it is only usable for people who spend a lot of time writing their own scripts or SQL queries to get information that can be used to make decisions.
We recognise that and had discussed the CGI WUI page as mentioned above.
Was this here on the list or elsewhere?
Adding another alternative to vnstat will result in us having to maintain two things that roughly do the same - or where one’s features are a subset of the other one.
I think Jon or Erik need to comment on any comparison with vnstat.
I would like to avoid merging this and then see it abandoned because it is too complicated to use. We already have a lot of virtually unmaintained stuff around and I would really like to drop a lot of it, although I know that there are a few people actually using it.
Jon and Erik should comment on the complication of use or not. Jon especially has done quite a bit of work with it, installing the .ipfire package versions that I have created. I intend to also have a play around with it but for me it is more some playful fun as I just have a home network with only myself as the user and admin.
Thoughts? Best, -Michael
Co-authored-by: Jon Murphy Co-authored-by: Erik Kapfer ummeegge@ipfire.org
Signed-off-by: Adolf Belka adolf.belka@ipfire.org
config/backup/includes/pmacct | 2 + config/pmacct/pmacct.conf | 24 +++++ config/pmacct/pmacct.init | 57 +++++++++++ config/pmacct/pmacct_memory_example.conf | 24 +++++ config/pmacct/pmacct_sqlite3_example.conf | 25 +++++ config/rootfiles/packages/pmacct | 111 ++++++++++++++++++++++ lfs/pmacct | 107 +++++++++++++++++++++ make.sh | 2 + src/paks/pmacct/install.sh | 36 +++++++ src/paks/pmacct/uninstall.sh | 33 +++++++ src/paks/pmacct/update.sh | 26 +++++ 11 files changed, 447 insertions(+) create mode 100644 config/backup/includes/pmacct create mode 100644 config/pmacct/pmacct.conf create mode 100644 config/pmacct/pmacct.init create mode 100644 config/pmacct/pmacct_memory_example.conf create mode 100644 config/pmacct/pmacct_sqlite3_example.conf create mode 100644 config/rootfiles/packages/pmacct create mode 100644 lfs/pmacct create mode 100644 src/paks/pmacct/install.sh create mode 100644 src/paks/pmacct/uninstall.sh create mode 100644 src/paks/pmacct/update.sh
diff --git a/config/backup/includes/pmacct b/config/backup/includes/pmacct new file mode 100644 index 000000000..4359067ad --- /dev/null +++ b/config/backup/includes/pmacct @@ -0,0 +1,2 @@ +/etc/pmacct/ +/var/spool/pmacct/ diff --git a/config/pmacct/pmacct.conf b/config/pmacct/pmacct.conf new file mode 100644 index 000000000..3c1d47efa --- /dev/null +++ b/config/pmacct/pmacct.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip \ No newline at end of file diff --git a/config/pmacct/pmacct.init b/config/pmacct/pmacct.init new file mode 100644 index 000000000..31f9ce49b --- /dev/null +++ b/config/pmacct/pmacct.init @@ -0,0 +1,57 @@ +#!/bin/bash
+# Begin $rc_base/init.d/pmacct
+# Starts and stops pmacct daemon +# Date: 2021-02-25 12:25:30 (Thu, 25 Feb 2021)
+# Locations +CONF="/etc/pmacct/pmacct.conf" +CONTENT="/var/pmacct" +# Pmacctd binary +PMACCT="/usr/sbin/pmacctd" +# Pmacctd start options '-f' for the configuration file +OPT="-f ${CONF}" +DESC="pmacct daemon" +# Pmacctd syslog facilitiy +#SYSLOG="local1"
+. /etc/sysconfig/rc +. $rc_functions
+case "$1" in
- start)
boot_mesg "Starting the ${DESC}... "
loadproc ${PMACCT} ${OPT}
sleep 2
if P=$(pgrep -fl pmacctd); then
echo -e "\n${P}"
else
echo -e "\n${PMACCT} is not running... "
fi
;;
- stop)
boot_mesg "Stopping the ${DESC}... "
killproc ${PMACCT}
;;
- restart)
$0 stop
sleep 1
$0 start
;;
- status)
statusproc ${PMACCT}
;;
- *)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
+esac
+# End $rc_base/init.d/pmacct
diff --git a/config/pmacct/pmacct_memory_example.conf b/config/pmacct/pmacct_memory_example.conf new file mode 100644 index 000000000..0babdaaf7 --- /dev/null +++ b/config/pmacct/pmacct_memory_example.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip diff --git a/config/pmacct/pmacct_sqlite3_example.conf b/config/pmacct/pmacct_sqlite3_example.conf new file mode 100644 index 000000000..7fb996408 --- /dev/null +++ b/config/pmacct/pmacct_sqlite3_example.conf @@ -0,0 +1,25 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin2" plugin configuration +! +plugins: sqlite3[plugin2]
+sql_db[plugin2]: /var/spool/pmacct/pmacct_sqlitev1.db +sql_table_version[plugin2]: 1 +sql_history[plugin2]: 5m +sql_refresh_time[plugin2]: 300 +sql_history_roundoff[plugin2]: m +sql_dont_try_update: true
+aggregate[plugin2]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin2]: ip diff --git a/config/rootfiles/packages/pmacct b/config/rootfiles/packages/pmacct new file mode 100644 index 000000000..fed83ee67 --- /dev/null +++ b/config/rootfiles/packages/pmacct @@ -0,0 +1,111 @@ +etc/pmacct +etc/pmacct/pmacct.conf +etc/pmacct/pmacct_memory_example.conf +etc/pmacct/pmacct_sqlite3_example.conf +etc/rc.d/init.d/pmacct +usr/bin/pmacct +#usr/lib/pmacct +#usr/lib/pmacct/examples +#usr/lib/pmacct/examples/custom +#usr/lib/pmacct/examples/custom/libcustom.la +#usr/lib/pmacct/examples/lg +#usr/lib/pmacct/examples/lg/pmbgp +usr/sbin/nfacctd +usr/sbin/pmacctd +usr/sbin/pmbgpd +usr/sbin/pmbmpd +usr/sbin/pmtelemetryd +usr/sbin/sfacctd +usr/share/pmacct +#usr/share/pmacct/CONFIG-KEYS +#usr/share/pmacct/FAQS +#usr/share/pmacct/QUICKSTART +#usr/share/pmacct/UPGRADE +#usr/share/pmacct/docs +#usr/share/pmacct/docs/IPFIX +#usr/share/pmacct/docs/LOOKING_GLASS_FORMAT +#usr/share/pmacct/docs/MSGLOG_DUMP_FORMATS +#usr/share/pmacct/docs/SIGNALS +#usr/share/pmacct/docs/TRIGGER_VARS +#usr/share/pmacct/examples +usr/share/pmacct/examples/allow.lst.example +#usr/share/pmacct/examples/amqp +usr/share/pmacct/examples/amqp/amqp_receiver.py +#usr/share/pmacct/examples/avro +usr/share/pmacct/examples/avro/avro_file_decoder.py +usr/share/pmacct/examples/bgp_agent.map.example +usr/share/pmacct/examples/bgp_md5.lst.example +usr/share/pmacct/examples/bgp_xconnects.map.example +usr/share/pmacct/examples/flow_to_rd.map.example +#usr/share/pmacct/examples/kafka +usr/share/pmacct/examples/kafka/kafka_consumer.py +#usr/share/pmacct/examples/lg +usr/share/pmacct/examples/lg/pmbgp.py +usr/share/pmacct/examples/networks.lst.example +usr/share/pmacct/examples/nfacctd-sql.conf.example +usr/share/pmacct/examples/pcap_interfaces.map.example +usr/share/pmacct/examples/peers.map.example +usr/share/pmacct/examples/pmacctd-imt.conf.example +usr/share/pmacct/examples/pmacctd-multiple-plugins.conf.example +usr/share/pmacct/examples/pmacctd-sql.conf.example +usr/share/pmacct/examples/pmacctd-sqlite3.conf.example +usr/share/pmacct/examples/ports.lst.example +usr/share/pmacct/examples/pretag.map.example +usr/share/pmacct/examples/primitives.lst.example +usr/share/pmacct/examples/probe_netflow.conf.example +usr/share/pmacct/examples/probe_sflow.conf.example +usr/share/pmacct/examples/sampling.map.example +usr/share/pmacct/examples/tee_receivers.lst.example +#usr/share/pmacct/sql +#usr/share/pmacct/sql/README.64bit +#usr/share/pmacct/sql/README.GeoIP +#usr/share/pmacct/sql/README.IPv6 +#usr/share/pmacct/sql/README.cos +#usr/share/pmacct/sql/README.custom_primitives +#usr/share/pmacct/sql/README.etype +#usr/share/pmacct/sql/README.export_proto +#usr/share/pmacct/sql/README.iface +#usr/share/pmacct/sql/README.label +#usr/share/pmacct/sql/README.mask +#usr/share/pmacct/sql/README.mpls +#usr/share/pmacct/sql/README.mysql +#usr/share/pmacct/sql/README.nat +#usr/share/pmacct/sql/README.pgsql +#usr/share/pmacct/sql/README.sampling +usr/share/pmacct/sql/README.sqlite3 +#usr/share/pmacct/sql/README.tag2 +#usr/share/pmacct/sql/README.timestamp +#usr/share/pmacct/sql/README.tunnel +#usr/share/pmacct/sql/pmacct-create-db.pgsql +#usr/share/pmacct/sql/pmacct-create-db_bgp_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v2.mysql +#usr/share/pmacct/sql/pmacct-create-db_v3.mysql +#usr/share/pmacct/sql/pmacct-create-db_v4.mysql +#usr/share/pmacct/sql/pmacct-create-db_v5.mysql +#usr/share/pmacct/sql/pmacct-create-db_v6.mysql +#usr/share/pmacct/sql/pmacct-create-db_v7.mysql +#usr/share/pmacct/sql/pmacct-create-db_v8.mysql +#usr/share/pmacct/sql/pmacct-create-db_v9.mysql +#usr/share/pmacct/sql/pmacct-create-table_bgp_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_bgp_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v2.pgsql +usr/share/pmacct/sql/pmacct-create-table_v2.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v3.pgsql +usr/share/pmacct/sql/pmacct-create-table_v3.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v4.pgsql +usr/share/pmacct/sql/pmacct-create-table_v4.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v5.pgsql +usr/share/pmacct/sql/pmacct-create-table_v5.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v6.pgsql +usr/share/pmacct/sql/pmacct-create-table_v6.sqlite3 +usr/share/pmacct/sql/pmacct-create-table_v7.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v7_v8.pgsql +usr/share/pmacct/sql/pmacct-create-table_v8.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v9.pgsql +usr/share/pmacct/sql/pmacct-create-table_v9.sqlite3 +#usr/share/pmacct/sql/pmacct-grant-db.mysql +var/ipfire/backup/addons/includes/pmacct +var/spool/pmacct diff --git a/lfs/pmacct b/lfs/pmacct new file mode 100644 index 000000000..c6cc5270e --- /dev/null +++ b/lfs/pmacct @@ -0,0 +1,107 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2019 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################
+############################################################################### +# Definitions +###############################################################################
+include Config
+VER = 1.7.6
+THISAPP = pmacct-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP) +PROG = pmacct +PAK_VER = 1
+DEPS = libcdada
+############################################################################### +# Top-level Rules +###############################################################################
+objects = $(DL_FILE)
+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
+$(DL_FILE)_MD5 = b829cfdc394d5acd6d32fb23de181a6b
+install : $(TARGET)
+check : $(patsubst %,$(DIR_CHK)/%,$(objects))
+download :$(patsubst %,$(DIR_DL)/%,$(objects))
+md5 : $(subst %,%_MD5,$(objects))
+dist:
- @$(PAK)
+############################################################################### +# Downloading, checking, md5sum +###############################################################################
+$(patsubst %,$(DIR_CHK)/%,$(objects)) :
- @$(CHECK)
+$(patsubst %,$(DIR_DL)/%,$(objects)) :
- @$(LOAD)
+$(subst %,%_MD5,$(objects)) :
- @$(MD5)
+############################################################################### +# Installation Details +###############################################################################
+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
- @$(PREBUILD)
- @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE)
- cd $(DIR_APP) && ./autogen.sh
- cd $(DIR_APP) && ./configure \
--prefix=/usr \
--sysconfdir=/etc/pmacct \
--enable-sqlite3 \
--enable-l2 \
--enable-jansson
- # Add directory for plugins and pipes
- -mkdir -v /var/spool/pmacct
- # Add configuration folder and copy conf files to it
- -mkdir -v /etc/pmacct
- cp -vf $(DIR_CONF)/pmacct/*.conf /etc/pmacct
- # Copy initscript to dir
- cp -vf $(DIR_CONF)/pmacct/pmacct.init /etc/rc.d/init.d/pmacct
- chmod 754 /etc/rc.d/init.d/pmacct
- chown root:root /etc/rc.d/init.d/pmacct
- cd $(DIR_APP) && make $(MAKETUNING)
- cd $(DIR_APP) && make install
- # Install backup definition
- install -v -m 644 ${DIR_SRC}/config/backup/includes/pmacct \
/var/ipfire/backup/addons/includes/pmacct
- @rm -rf $(DIR_APP)
- @$(POSTBUILD)
diff --git a/make.sh b/make.sh index 43cccaca0..d42c9b1e2 100755 --- a/make.sh +++ b/make.sh @@ -1646,6 +1646,8 @@ buildipfire() { lfsmake2 ncdu lfsmake2 lshw lfsmake2 socat
- lfsmake2 libcdada
- lfsmake2 pmacct
}
buildinstaller() { diff --git a/src/paks/pmacct/install.sh b/src/paks/pmacct/install.sh new file mode 100644 index 000000000..8178cc7bc --- /dev/null +++ b/src/paks/pmacct/install.sh @@ -0,0 +1,36 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh
+extract_files +restore_backup ${NAME}
+# Add symlinks for runlevels +ln -s ../init.d/${NAME} /etc/rc.d/rc0.d/K85${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc3.d/S50${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc6.d/K85${NAME} +start_service --background ${NAME}
+# EOF diff --git a/src/paks/pmacct/uninstall.sh b/src/paks/pmacct/uninstall.sh new file mode 100644 index 000000000..51631e8c1 --- /dev/null +++ b/src/paks/pmacct/uninstall.sh @@ -0,0 +1,33 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh
+stop_service ${NAME} +make_backup ${NAME} +remove_files
+# Delete symlinks +rm -rfv /etc/rc.d/rc?.d/???${NAME}
+# EOF diff --git a/src/paks/pmacct/update.sh b/src/paks/pmacct/update.sh new file mode 100644 index 000000000..89c40d0d7 --- /dev/null +++ b/src/paks/pmacct/update.sh @@ -0,0 +1,26 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh +./uninstall.sh
+./install.sh
2.30.1
Hi Michael,
On 18/03/2021 19:25, Michael Tremer wrote:
Hey,
On 18 Mar 2021, at 13:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 18/03/2021 14:07, Michael Tremer wrote:
Hello, I will split this email into two parts: The technical part:
- The initscript isn’t based on our template and uses sleep to check if starting a daemon has been successful. That should not be necessary and will potentially slow down the boot process.
I will have another look at the initscript and see if it can use the standard IPFire template.
Yes, it is not too different from your version:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/initscripts/system/tem...
There isn’t much documentation around it but there are loads of other scripts that will be helpful to figure out any convention or boilerplate code to copy.
- I am not very sure if the example configuration files are helpful. Maybe someone can clear this up a little bit for me what is the rationale is behind them.
The idea was that the two simplest config example scripts were provided that give a user the ability to get data into memory or into a sqlite file. The actual conf file provided is a copy of the memory version. These give a basic starting point that people can then add additional info to be recorded etc.
Does anybody want to use anything else than SQLite in the context of IPFire?
I think we weren't sure and the memory and sqlite options were the easiest to get going and give real data. There are also mysql and postgresql options but I think we need to see what the interest is.
- There are loads of daemons being shipped and only one is being used.
I think Jon or Erik need to comment on this question.
- How much load does this generate? How large is the captured data?
Jon needs to provide this feedback.
Then there is the other part:
On 8 Mar 2021, at 11:57, Adolf Belka (ipfire) adolf.belka@ipfire.org wrote:
From: Adolf Belka adolf.belka@ipfire.org
- What is it? pmacct is a monitoring tool for network management tasks. Data collected can be used for analysis and troubleshooting purposes to maintain the health of the network. pmacct can collect, replicate and export network information. It can cache in memory tables, store persistently to SQLite3 and output to flat-files like CSV, formatted, and JSON.
Okay.
- Why is it needed? To monitor data usage (IP-based or MAC-based data accounting) down to the client level. Net-Traffic will monitor traffic for the entire RED, GREEN, etc. networks, but it cannot pinpoint which client is using lots of data. Connections will take a snapshot but not show day by day sums. pmacct can help admins keep tabs on users that use too much data.
I agree with the goal of it. But how does this add-on achieve that?
Jon should comment on this as he had this use case and can give feedback how the data has helped him in his investigations.
The data is being written to a (usually binary) file which cannot be visualised in the web UI. Does this not want a CGI script that allows people to see what is happening?
We all thought that this would ultimately need a CGI script to set up and visualise the data. We believed it was worth getting this running in actual use as a CLI tool to start with and then get more feedback on the best items that would need to be captured in the CGI script. The tool has a lot of capabilities and I think it will not be possible to do everything that it can do in a WUI page. So some form of file or db offload will need to be available anyway but the items identified as being commonly required would be the ones to be developed into a CGI WUI page.
Sending manual SQL queries probably isn’t much fun :)
- What are the use cases? An ISP may implement data caps and if the limit is over-run then you have to pay for every additional xxGB of data used. Typical charges can be around $10 per 50GB. With pmacct you can identify the high users and take action, hopefully before the limit is breached.
- This is being introduced as a command line only tool. However, at a later date, if it is useful to enough additional users a WUI page could be developed as discussed in the development mailing list https://lists.ipfire.org/pipermail/development/2021-January/009174.html
I have been thinking about this for a while now and would say that I can approve merging this as a CLI tool. But it would not really enhance IPFire’s functionality because it is only usable for people who spend a lot of time writing their own scripts or SQL queries to get information that can be used to make decisions.
We recognise that and had discussed the CGI WUI page as mentioned above.
Was this here on the list or elsewhere?
This was in a private conversation on the IPFire Community forum. We thought it best to have all our back and forth discussions between ourselves to start with. We were all finding our feet with it as well and we thought it best not to swamp the development list with our musings and questions to each other. If you want to see all the discussion history, then we can add you to the members of that conversation. If the addon gets accepted then any future discussions would be done on the Development mailing list to get everyones input.
Adding another alternative to vnstat will result in us having to maintain two things that roughly do the same - or where one’s features are a subset of the other one.
I think Jon or Erik need to comment on any comparison with vnstat.
I would like to avoid merging this and then see it abandoned because it is too complicated to use. We already have a lot of virtually unmaintained stuff around and I would really like to drop a lot of it, although I know that there are a few people actually using it.
Jon and Erik should comment on the complication of use or not. Jon especially has done quite a bit of work with it, installing the .ipfire package versions that I have created. I intend to also have a play around with it but for me it is more some playful fun as I just have a home network with only myself as the user and admin.
Thoughts? Best, -Michael
Co-authored-by: Jon Murphy Co-authored-by: Erik Kapfer ummeegge@ipfire.org
Signed-off-by: Adolf Belka adolf.belka@ipfire.org
config/backup/includes/pmacct | 2 + config/pmacct/pmacct.conf | 24 +++++ config/pmacct/pmacct.init | 57 +++++++++++ config/pmacct/pmacct_memory_example.conf | 24 +++++ config/pmacct/pmacct_sqlite3_example.conf | 25 +++++ config/rootfiles/packages/pmacct | 111 ++++++++++++++++++++++ lfs/pmacct | 107 +++++++++++++++++++++ make.sh | 2 + src/paks/pmacct/install.sh | 36 +++++++ src/paks/pmacct/uninstall.sh | 33 +++++++ src/paks/pmacct/update.sh | 26 +++++ 11 files changed, 447 insertions(+) create mode 100644 config/backup/includes/pmacct create mode 100644 config/pmacct/pmacct.conf create mode 100644 config/pmacct/pmacct.init create mode 100644 config/pmacct/pmacct_memory_example.conf create mode 100644 config/pmacct/pmacct_sqlite3_example.conf create mode 100644 config/rootfiles/packages/pmacct create mode 100644 lfs/pmacct create mode 100644 src/paks/pmacct/install.sh create mode 100644 src/paks/pmacct/uninstall.sh create mode 100644 src/paks/pmacct/update.sh
diff --git a/config/backup/includes/pmacct b/config/backup/includes/pmacct new file mode 100644 index 000000000..4359067ad --- /dev/null +++ b/config/backup/includes/pmacct @@ -0,0 +1,2 @@ +/etc/pmacct/ +/var/spool/pmacct/ diff --git a/config/pmacct/pmacct.conf b/config/pmacct/pmacct.conf new file mode 100644 index 000000000..3c1d47efa --- /dev/null +++ b/config/pmacct/pmacct.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip \ No newline at end of file diff --git a/config/pmacct/pmacct.init b/config/pmacct/pmacct.init new file mode 100644 index 000000000..31f9ce49b --- /dev/null +++ b/config/pmacct/pmacct.init @@ -0,0 +1,57 @@ +#!/bin/bash
+# Begin $rc_base/init.d/pmacct
+# Starts and stops pmacct daemon +# Date: 2021-02-25 12:25:30 (Thu, 25 Feb 2021)
+# Locations +CONF="/etc/pmacct/pmacct.conf" +CONTENT="/var/pmacct" +# Pmacctd binary +PMACCT="/usr/sbin/pmacctd" +# Pmacctd start options '-f' for the configuration file +OPT="-f ${CONF}" +DESC="pmacct daemon" +# Pmacctd syslog facilitiy +#SYSLOG="local1"
+. /etc/sysconfig/rc +. $rc_functions
+case "$1" in
- start)
boot_mesg "Starting the ${DESC}... "
loadproc ${PMACCT} ${OPT}
sleep 2
if P=$(pgrep -fl pmacctd); then
echo -e "\n${P}"
else
echo -e "\n${PMACCT} is not running... "
fi
;;
- stop)
boot_mesg "Stopping the ${DESC}... "
killproc ${PMACCT}
;;
- restart)
$0 stop
sleep 1
$0 start
;;
- status)
statusproc ${PMACCT}
;;
- *)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
+esac
+# End $rc_base/init.d/pmacct
diff --git a/config/pmacct/pmacct_memory_example.conf b/config/pmacct/pmacct_memory_example.conf new file mode 100644 index 000000000..0babdaaf7 --- /dev/null +++ b/config/pmacct/pmacct_memory_example.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip diff --git a/config/pmacct/pmacct_sqlite3_example.conf b/config/pmacct/pmacct_sqlite3_example.conf new file mode 100644 index 000000000..7fb996408 --- /dev/null +++ b/config/pmacct/pmacct_sqlite3_example.conf @@ -0,0 +1,25 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin2" plugin configuration +! +plugins: sqlite3[plugin2]
+sql_db[plugin2]: /var/spool/pmacct/pmacct_sqlitev1.db +sql_table_version[plugin2]: 1 +sql_history[plugin2]: 5m +sql_refresh_time[plugin2]: 300 +sql_history_roundoff[plugin2]: m +sql_dont_try_update: true
+aggregate[plugin2]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin2]: ip diff --git a/config/rootfiles/packages/pmacct b/config/rootfiles/packages/pmacct new file mode 100644 index 000000000..fed83ee67 --- /dev/null +++ b/config/rootfiles/packages/pmacct @@ -0,0 +1,111 @@ +etc/pmacct +etc/pmacct/pmacct.conf +etc/pmacct/pmacct_memory_example.conf +etc/pmacct/pmacct_sqlite3_example.conf +etc/rc.d/init.d/pmacct +usr/bin/pmacct +#usr/lib/pmacct +#usr/lib/pmacct/examples +#usr/lib/pmacct/examples/custom +#usr/lib/pmacct/examples/custom/libcustom.la +#usr/lib/pmacct/examples/lg +#usr/lib/pmacct/examples/lg/pmbgp +usr/sbin/nfacctd +usr/sbin/pmacctd +usr/sbin/pmbgpd +usr/sbin/pmbmpd +usr/sbin/pmtelemetryd +usr/sbin/sfacctd +usr/share/pmacct +#usr/share/pmacct/CONFIG-KEYS +#usr/share/pmacct/FAQS +#usr/share/pmacct/QUICKSTART +#usr/share/pmacct/UPGRADE +#usr/share/pmacct/docs +#usr/share/pmacct/docs/IPFIX +#usr/share/pmacct/docs/LOOKING_GLASS_FORMAT +#usr/share/pmacct/docs/MSGLOG_DUMP_FORMATS +#usr/share/pmacct/docs/SIGNALS +#usr/share/pmacct/docs/TRIGGER_VARS +#usr/share/pmacct/examples +usr/share/pmacct/examples/allow.lst.example +#usr/share/pmacct/examples/amqp +usr/share/pmacct/examples/amqp/amqp_receiver.py +#usr/share/pmacct/examples/avro +usr/share/pmacct/examples/avro/avro_file_decoder.py +usr/share/pmacct/examples/bgp_agent.map.example +usr/share/pmacct/examples/bgp_md5.lst.example +usr/share/pmacct/examples/bgp_xconnects.map.example +usr/share/pmacct/examples/flow_to_rd.map.example +#usr/share/pmacct/examples/kafka +usr/share/pmacct/examples/kafka/kafka_consumer.py +#usr/share/pmacct/examples/lg +usr/share/pmacct/examples/lg/pmbgp.py +usr/share/pmacct/examples/networks.lst.example +usr/share/pmacct/examples/nfacctd-sql.conf.example +usr/share/pmacct/examples/pcap_interfaces.map.example +usr/share/pmacct/examples/peers.map.example +usr/share/pmacct/examples/pmacctd-imt.conf.example +usr/share/pmacct/examples/pmacctd-multiple-plugins.conf.example +usr/share/pmacct/examples/pmacctd-sql.conf.example +usr/share/pmacct/examples/pmacctd-sqlite3.conf.example +usr/share/pmacct/examples/ports.lst.example +usr/share/pmacct/examples/pretag.map.example +usr/share/pmacct/examples/primitives.lst.example +usr/share/pmacct/examples/probe_netflow.conf.example +usr/share/pmacct/examples/probe_sflow.conf.example +usr/share/pmacct/examples/sampling.map.example +usr/share/pmacct/examples/tee_receivers.lst.example +#usr/share/pmacct/sql +#usr/share/pmacct/sql/README.64bit +#usr/share/pmacct/sql/README.GeoIP +#usr/share/pmacct/sql/README.IPv6 +#usr/share/pmacct/sql/README.cos +#usr/share/pmacct/sql/README.custom_primitives +#usr/share/pmacct/sql/README.etype +#usr/share/pmacct/sql/README.export_proto +#usr/share/pmacct/sql/README.iface +#usr/share/pmacct/sql/README.label +#usr/share/pmacct/sql/README.mask +#usr/share/pmacct/sql/README.mpls +#usr/share/pmacct/sql/README.mysql +#usr/share/pmacct/sql/README.nat +#usr/share/pmacct/sql/README.pgsql +#usr/share/pmacct/sql/README.sampling +usr/share/pmacct/sql/README.sqlite3 +#usr/share/pmacct/sql/README.tag2 +#usr/share/pmacct/sql/README.timestamp +#usr/share/pmacct/sql/README.tunnel +#usr/share/pmacct/sql/pmacct-create-db.pgsql +#usr/share/pmacct/sql/pmacct-create-db_bgp_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v2.mysql +#usr/share/pmacct/sql/pmacct-create-db_v3.mysql +#usr/share/pmacct/sql/pmacct-create-db_v4.mysql +#usr/share/pmacct/sql/pmacct-create-db_v5.mysql +#usr/share/pmacct/sql/pmacct-create-db_v6.mysql +#usr/share/pmacct/sql/pmacct-create-db_v7.mysql +#usr/share/pmacct/sql/pmacct-create-db_v8.mysql +#usr/share/pmacct/sql/pmacct-create-db_v9.mysql +#usr/share/pmacct/sql/pmacct-create-table_bgp_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_bgp_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v2.pgsql +usr/share/pmacct/sql/pmacct-create-table_v2.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v3.pgsql +usr/share/pmacct/sql/pmacct-create-table_v3.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v4.pgsql +usr/share/pmacct/sql/pmacct-create-table_v4.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v5.pgsql +usr/share/pmacct/sql/pmacct-create-table_v5.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v6.pgsql +usr/share/pmacct/sql/pmacct-create-table_v6.sqlite3 +usr/share/pmacct/sql/pmacct-create-table_v7.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v7_v8.pgsql +usr/share/pmacct/sql/pmacct-create-table_v8.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v9.pgsql +usr/share/pmacct/sql/pmacct-create-table_v9.sqlite3 +#usr/share/pmacct/sql/pmacct-grant-db.mysql +var/ipfire/backup/addons/includes/pmacct +var/spool/pmacct diff --git a/lfs/pmacct b/lfs/pmacct new file mode 100644 index 000000000..c6cc5270e --- /dev/null +++ b/lfs/pmacct @@ -0,0 +1,107 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2019 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################
+############################################################################### +# Definitions +###############################################################################
+include Config
+VER = 1.7.6
+THISAPP = pmacct-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP) +PROG = pmacct +PAK_VER = 1
+DEPS = libcdada
+############################################################################### +# Top-level Rules +###############################################################################
+objects = $(DL_FILE)
+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
+$(DL_FILE)_MD5 = b829cfdc394d5acd6d32fb23de181a6b
+install : $(TARGET)
+check : $(patsubst %,$(DIR_CHK)/%,$(objects))
+download :$(patsubst %,$(DIR_DL)/%,$(objects))
+md5 : $(subst %,%_MD5,$(objects))
+dist:
- @$(PAK)
+############################################################################### +# Downloading, checking, md5sum +###############################################################################
+$(patsubst %,$(DIR_CHK)/%,$(objects)) :
- @$(CHECK)
+$(patsubst %,$(DIR_DL)/%,$(objects)) :
- @$(LOAD)
+$(subst %,%_MD5,$(objects)) :
- @$(MD5)
+############################################################################### +# Installation Details +###############################################################################
+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))
- @$(PREBUILD)
- @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE)
- cd $(DIR_APP) && ./autogen.sh
- cd $(DIR_APP) && ./configure \
--prefix=/usr \
--sysconfdir=/etc/pmacct \
--enable-sqlite3 \
--enable-l2 \
--enable-jansson
- # Add directory for plugins and pipes
- -mkdir -v /var/spool/pmacct
- # Add configuration folder and copy conf files to it
- -mkdir -v /etc/pmacct
- cp -vf $(DIR_CONF)/pmacct/*.conf /etc/pmacct
- # Copy initscript to dir
- cp -vf $(DIR_CONF)/pmacct/pmacct.init /etc/rc.d/init.d/pmacct
- chmod 754 /etc/rc.d/init.d/pmacct
- chown root:root /etc/rc.d/init.d/pmacct
- cd $(DIR_APP) && make $(MAKETUNING)
- cd $(DIR_APP) && make install
- # Install backup definition
- install -v -m 644 ${DIR_SRC}/config/backup/includes/pmacct \
/var/ipfire/backup/addons/includes/pmacct
- @rm -rf $(DIR_APP)
- @$(POSTBUILD)
diff --git a/make.sh b/make.sh index 43cccaca0..d42c9b1e2 100755 --- a/make.sh +++ b/make.sh @@ -1646,6 +1646,8 @@ buildipfire() { lfsmake2 ncdu lfsmake2 lshw lfsmake2 socat
- lfsmake2 libcdada
- lfsmake2 pmacct
}
buildinstaller() { diff --git a/src/paks/pmacct/install.sh b/src/paks/pmacct/install.sh new file mode 100644 index 000000000..8178cc7bc --- /dev/null +++ b/src/paks/pmacct/install.sh @@ -0,0 +1,36 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh
+extract_files +restore_backup ${NAME}
+# Add symlinks for runlevels +ln -s ../init.d/${NAME} /etc/rc.d/rc0.d/K85${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc3.d/S50${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc6.d/K85${NAME} +start_service --background ${NAME}
+# EOF diff --git a/src/paks/pmacct/uninstall.sh b/src/paks/pmacct/uninstall.sh new file mode 100644 index 000000000..51631e8c1 --- /dev/null +++ b/src/paks/pmacct/uninstall.sh @@ -0,0 +1,33 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh
+stop_service ${NAME} +make_backup ${NAME} +remove_files
+# Delete symlinks +rm -rfv /etc/rc.d/rc?.d/???${NAME}
+# EOF diff --git a/src/paks/pmacct/update.sh b/src/paks/pmacct/update.sh new file mode 100644 index 000000000..89c40d0d7 --- /dev/null +++ b/src/paks/pmacct/update.sh @@ -0,0 +1,26 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh +./uninstall.sh
+./install.sh
2.30.1
Hi Michael,
On 18/03/2021 19:43, Adolf Belka wrote:
Hi Michael,
On 18/03/2021 19:25, Michael Tremer wrote:
Hey,
On 18 Mar 2021, at 13:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 18/03/2021 14:07, Michael Tremer wrote:
Hello, I will split this email into two parts: The technical part:
- The initscript isn’t based on our template and uses sleep to
check if starting a daemon has been successful. That should not be necessary and will potentially slow down the boot process.
I will have another look at the initscript and see if it can use the standard IPFire template.
Yes, it is not too different from your version:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/initscripts/system/tem...
There isn’t much documentation around it but there are loads of other scripts that will be helpful to figure out any convention or boilerplate code to copy.
- I am not very sure if the example configuration files are
helpful. Maybe someone can clear this up a little bit for me what is the rationale is behind them.
The idea was that the two simplest config example scripts were provided that give a user the ability to get data into memory or into a sqlite file. The actual conf file provided is a copy of the memory version. These give a basic starting point that people can then add additional info to be recorded etc.
Does anybody want to use anything else than SQLite in the context of IPFire?
I think we weren't sure and the memory and sqlite options were the easiest to get going and give real data. There are also mysql and postgresql options but I think we need to see what the interest is.
- There are loads of daemons being shipped and only one is being used.
I looked through our conversations and it looks like these were left there because we weren't sure if they would be used or not.
pmacctd is the daemon being used by Jon. pmacct is the frontend to pmacctd when being used in memory mode.
For the others I looked in the pmacct website and found the following:-
I looked through our conversations and it looks like these were left there because we weren't sure if they would be used or not.
pmacctd is the daemon being used by Jon. pmacct is the frontend to pmacctd when being used in memory mode.
For the others I looked in the pmacct website and found the following:-
'pmacctd' is the libpcap-based IPv4/IPv6 accounting daemon
'nfacctd' is the NetFlow (versions supported NetFlow v1, v5 and v9) and IPFIX accounting daemon
'sfacctd' is the sFlow v2/v4 and v5 accounting daemon
'uacctd' is the Linux NFLOG-based accounting daemon (historically, it was using ULOG, hence its name)
'pmtelemetryd' is the Streaming Telemetry collector, where "Streaming telemetry [ .. ] data can be used for analysis and troubleshooting purposes to maintain the health of the network. This is achieved by leveraging the capabilities of machine-to-machine communication. [ .. ]" (quoting Cisco IOS-XR Telemetry Configuration Guide at the time of this writing)
'pmbgpd' is the pmacct BGP collector and 'pmbmpd'is the pmacct BMP collector.
I don't know what most of that means but it looks to be quite different to what the addon was created for, so I would propose that I re-do the rootfiles and remove those daemons from the install. If in future there is demand for others we can add them back in with an update patch. I will do that together with the other changes already discussed.
Regards,
Adolf.
I think Jon or Erik need to comment on this question.
- How much load does this generate? How large is the captured data?
Jon needs to provide this feedback.
Then there is the other part:
On 8 Mar 2021, at 11:57, Adolf Belka (ipfire) adolf.belka@ipfire.org wrote:
From: Adolf Belka adolf.belka@ipfire.org
- What is it?
pmacct is a monitoring tool for network management tasks. Data collected can be used for analysis and troubleshooting purposes to maintain the health of the network. pmacct can collect, replicate and export network information. It can cache in memory tables, store persistently to SQLite3 and output to flat-files like CSV, formatted, and JSON.
Okay.
- Why is it needed?
To monitor data usage (IP-based or MAC-based data accounting) down to the client level. Net-Traffic will monitor traffic for the entire RED, GREEN, etc. networks, but it cannot pinpoint which client is using lots of data. Connections will take a snapshot but not show day by day sums. pmacct can help admins keep tabs on users that use too much data.
I agree with the goal of it. But how does this add-on achieve that?
Jon should comment on this as he had this use case and can give feedback how the data has helped him in his investigations.
The data is being written to a (usually binary) file which cannot be visualised in the web UI. Does this not want a CGI script that allows people to see what is happening?
We all thought that this would ultimately need a CGI script to set up and visualise the data. We believed it was worth getting this running in actual use as a CLI tool to start with and then get more feedback on the best items that would need to be captured in the CGI script. The tool has a lot of capabilities and I think it will not be possible to do everything that it can do in a WUI page. So some form of file or db offload will need to be available anyway but the items identified as being commonly required would be the ones to be developed into a CGI WUI page.
Sending manual SQL queries probably isn’t much fun :)
- What are the use cases?
An ISP may implement data caps and if the limit is over-run then you have to pay for every additional xxGB of data used. Typical charges can be around $10 per 50GB. With pmacct you can identify the high users and take action, hopefully before the limit is breached.
- This is being introduced as a command line only tool. However,
at a later date, if it is useful to enough additional users a WUI page could be developed as discussed in the development mailing list https://lists.ipfire.org/pipermail/development/2021-January/009174.html
I have been thinking about this for a while now and would say that I can approve merging this as a CLI tool. But it would not really enhance IPFire’s functionality because it is only usable for people who spend a lot of time writing their own scripts or SQL queries to get information that can be used to make decisions.
We recognise that and had discussed the CGI WUI page as mentioned above.
Was this here on the list or elsewhere?
This was in a private conversation on the IPFire Community forum. We thought it best to have all our back and forth discussions between ourselves to start with. We were all finding our feet with it as well and we thought it best not to swamp the development list with our musings and questions to each other. If you want to see all the discussion history, then we can add you to the members of that conversation. If the addon gets accepted then any future discussions would be done on the Development mailing list to get everyones input.
Adding another alternative to vnstat will result in us having to maintain two things that roughly do the same - or where one’s features are a subset of the other one.
I think Jon or Erik need to comment on any comparison with vnstat.
I would like to avoid merging this and then see it abandoned because it is too complicated to use. We already have a lot of virtually unmaintained stuff around and I would really like to drop a lot of it, although I know that there are a few people actually using it.
Jon and Erik should comment on the complication of use or not. Jon especially has done quite a bit of work with it, installing the .ipfire package versions that I have created. I intend to also have a play around with it but for me it is more some playful fun as I just have a home network with only myself as the user and admin.
Thoughts? Best, -Michael
Co-authored-by: Jon Murphy Co-authored-by: Erik Kapfer ummeegge@ipfire.org
Signed-off-by: Adolf Belka adolf.belka@ipfire.org
config/backup/includes/pmacct | 2 + config/pmacct/pmacct.conf | 24 +++++ config/pmacct/pmacct.init | 57 +++++++++++ config/pmacct/pmacct_memory_example.conf | 24 +++++ config/pmacct/pmacct_sqlite3_example.conf | 25 +++++ config/rootfiles/packages/pmacct | 111 ++++++++++++++++++++++ lfs/pmacct | 107 +++++++++++++++++++++ make.sh | 2 + src/paks/pmacct/install.sh | 36 +++++++ src/paks/pmacct/uninstall.sh | 33 +++++++ src/paks/pmacct/update.sh | 26 +++++ 11 files changed, 447 insertions(+) create mode 100644 config/backup/includes/pmacct create mode 100644 config/pmacct/pmacct.conf create mode 100644 config/pmacct/pmacct.init create mode 100644 config/pmacct/pmacct_memory_example.conf create mode 100644 config/pmacct/pmacct_sqlite3_example.conf create mode 100644 config/rootfiles/packages/pmacct create mode 100644 lfs/pmacct create mode 100644 src/paks/pmacct/install.sh create mode 100644 src/paks/pmacct/uninstall.sh create mode 100644 src/paks/pmacct/update.sh
diff --git a/config/backup/includes/pmacct b/config/backup/includes/pmacct new file mode 100644 index 000000000..4359067ad --- /dev/null +++ b/config/backup/includes/pmacct @@ -0,0 +1,2 @@ +/etc/pmacct/ +/var/spool/pmacct/ diff --git a/config/pmacct/pmacct.conf b/config/pmacct/pmacct.conf new file mode 100644 index 000000000..3c1d47efa --- /dev/null +++ b/config/pmacct/pmacct.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip \ No newline at end of file diff --git a/config/pmacct/pmacct.init b/config/pmacct/pmacct.init new file mode 100644 index 000000000..31f9ce49b --- /dev/null +++ b/config/pmacct/pmacct.init @@ -0,0 +1,57 @@ +#!/bin/bash
+# Begin $rc_base/init.d/pmacct
+# Starts and stops pmacct daemon +# Date: 2021-02-25 12:25:30 (Thu, 25 Feb 2021)
+# Locations +CONF="/etc/pmacct/pmacct.conf" +CONTENT="/var/pmacct" +# Pmacctd binary +PMACCT="/usr/sbin/pmacctd" +# Pmacctd start options '-f' for the configuration file +OPT="-f ${CONF}" +DESC="pmacct daemon" +# Pmacctd syslog facilitiy +#SYSLOG="local1"
+. /etc/sysconfig/rc +. $rc_functions
+case "$1" in + start) + boot_mesg "Starting the ${DESC}... " + loadproc ${PMACCT} ${OPT} + sleep 2 + if P=$(pgrep -fl pmacctd); then + echo -e "\n${P}" + else + echo -e "\n${PMACCT} is not running... " + fi + ;;
+ stop) + boot_mesg "Stopping the ${DESC}... " + killproc ${PMACCT} + ;;
+ restart) + $0 stop + sleep 1 + $0 start + ;;
+ status) + statusproc ${PMACCT} + ;;
+ *) + echo "Usage: $0 {start|stop|restart|status}" + exit 1 + ;; +esac
+# End $rc_base/init.d/pmacct
diff --git a/config/pmacct/pmacct_memory_example.conf b/config/pmacct/pmacct_memory_example.conf new file mode 100644 index 000000000..0babdaaf7 --- /dev/null +++ b/config/pmacct/pmacct_memory_example.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip diff --git a/config/pmacct/pmacct_sqlite3_example.conf b/config/pmacct/pmacct_sqlite3_example.conf new file mode 100644 index 000000000..7fb996408 --- /dev/null +++ b/config/pmacct/pmacct_sqlite3_example.conf @@ -0,0 +1,25 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin2" plugin configuration +! +plugins: sqlite3[plugin2]
+sql_db[plugin2]: /var/spool/pmacct/pmacct_sqlitev1.db +sql_table_version[plugin2]: 1 +sql_history[plugin2]: 5m +sql_refresh_time[plugin2]: 300 +sql_history_roundoff[plugin2]: m +sql_dont_try_update: true
+aggregate[plugin2]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin2]: ip diff --git a/config/rootfiles/packages/pmacct b/config/rootfiles/packages/pmacct new file mode 100644 index 000000000..fed83ee67 --- /dev/null +++ b/config/rootfiles/packages/pmacct @@ -0,0 +1,111 @@ +etc/pmacct +etc/pmacct/pmacct.conf +etc/pmacct/pmacct_memory_example.conf +etc/pmacct/pmacct_sqlite3_example.conf +etc/rc.d/init.d/pmacct +usr/bin/pmacct +#usr/lib/pmacct +#usr/lib/pmacct/examples +#usr/lib/pmacct/examples/custom +#usr/lib/pmacct/examples/custom/libcustom.la +#usr/lib/pmacct/examples/lg +#usr/lib/pmacct/examples/lg/pmbgp +usr/sbin/nfacctd +usr/sbin/pmacctd +usr/sbin/pmbgpd +usr/sbin/pmbmpd +usr/sbin/pmtelemetryd +usr/sbin/sfacctd +usr/share/pmacct +#usr/share/pmacct/CONFIG-KEYS +#usr/share/pmacct/FAQS +#usr/share/pmacct/QUICKSTART +#usr/share/pmacct/UPGRADE +#usr/share/pmacct/docs +#usr/share/pmacct/docs/IPFIX +#usr/share/pmacct/docs/LOOKING_GLASS_FORMAT +#usr/share/pmacct/docs/MSGLOG_DUMP_FORMATS +#usr/share/pmacct/docs/SIGNALS +#usr/share/pmacct/docs/TRIGGER_VARS +#usr/share/pmacct/examples +usr/share/pmacct/examples/allow.lst.example +#usr/share/pmacct/examples/amqp +usr/share/pmacct/examples/amqp/amqp_receiver.py +#usr/share/pmacct/examples/avro +usr/share/pmacct/examples/avro/avro_file_decoder.py +usr/share/pmacct/examples/bgp_agent.map.example +usr/share/pmacct/examples/bgp_md5.lst.example +usr/share/pmacct/examples/bgp_xconnects.map.example +usr/share/pmacct/examples/flow_to_rd.map.example +#usr/share/pmacct/examples/kafka +usr/share/pmacct/examples/kafka/kafka_consumer.py +#usr/share/pmacct/examples/lg +usr/share/pmacct/examples/lg/pmbgp.py +usr/share/pmacct/examples/networks.lst.example +usr/share/pmacct/examples/nfacctd-sql.conf.example +usr/share/pmacct/examples/pcap_interfaces.map.example +usr/share/pmacct/examples/peers.map.example +usr/share/pmacct/examples/pmacctd-imt.conf.example +usr/share/pmacct/examples/pmacctd-multiple-plugins.conf.example +usr/share/pmacct/examples/pmacctd-sql.conf.example +usr/share/pmacct/examples/pmacctd-sqlite3.conf.example +usr/share/pmacct/examples/ports.lst.example +usr/share/pmacct/examples/pretag.map.example +usr/share/pmacct/examples/primitives.lst.example +usr/share/pmacct/examples/probe_netflow.conf.example +usr/share/pmacct/examples/probe_sflow.conf.example +usr/share/pmacct/examples/sampling.map.example +usr/share/pmacct/examples/tee_receivers.lst.example +#usr/share/pmacct/sql +#usr/share/pmacct/sql/README.64bit +#usr/share/pmacct/sql/README.GeoIP +#usr/share/pmacct/sql/README.IPv6 +#usr/share/pmacct/sql/README.cos +#usr/share/pmacct/sql/README.custom_primitives +#usr/share/pmacct/sql/README.etype +#usr/share/pmacct/sql/README.export_proto +#usr/share/pmacct/sql/README.iface +#usr/share/pmacct/sql/README.label +#usr/share/pmacct/sql/README.mask +#usr/share/pmacct/sql/README.mpls +#usr/share/pmacct/sql/README.mysql +#usr/share/pmacct/sql/README.nat +#usr/share/pmacct/sql/README.pgsql +#usr/share/pmacct/sql/README.sampling +usr/share/pmacct/sql/README.sqlite3 +#usr/share/pmacct/sql/README.tag2 +#usr/share/pmacct/sql/README.timestamp +#usr/share/pmacct/sql/README.tunnel +#usr/share/pmacct/sql/pmacct-create-db.pgsql +#usr/share/pmacct/sql/pmacct-create-db_bgp_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v2.mysql +#usr/share/pmacct/sql/pmacct-create-db_v3.mysql +#usr/share/pmacct/sql/pmacct-create-db_v4.mysql +#usr/share/pmacct/sql/pmacct-create-db_v5.mysql +#usr/share/pmacct/sql/pmacct-create-db_v6.mysql +#usr/share/pmacct/sql/pmacct-create-db_v7.mysql +#usr/share/pmacct/sql/pmacct-create-db_v8.mysql +#usr/share/pmacct/sql/pmacct-create-db_v9.mysql +#usr/share/pmacct/sql/pmacct-create-table_bgp_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_bgp_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v2.pgsql +usr/share/pmacct/sql/pmacct-create-table_v2.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v3.pgsql +usr/share/pmacct/sql/pmacct-create-table_v3.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v4.pgsql +usr/share/pmacct/sql/pmacct-create-table_v4.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v5.pgsql +usr/share/pmacct/sql/pmacct-create-table_v5.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v6.pgsql +usr/share/pmacct/sql/pmacct-create-table_v6.sqlite3 +usr/share/pmacct/sql/pmacct-create-table_v7.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v7_v8.pgsql +usr/share/pmacct/sql/pmacct-create-table_v8.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v9.pgsql +usr/share/pmacct/sql/pmacct-create-table_v9.sqlite3 +#usr/share/pmacct/sql/pmacct-grant-db.mysql +var/ipfire/backup/addons/includes/pmacct +var/spool/pmacct diff --git a/lfs/pmacct b/lfs/pmacct new file mode 100644 index 000000000..c6cc5270e --- /dev/null +++ b/lfs/pmacct @@ -0,0 +1,107 @@ +###############################################################################
+# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2019 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################
+###############################################################################
+# Definitions +###############################################################################
+include Config
+VER = 1.7.6
+THISAPP = pmacct-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP) +PROG = pmacct +PAK_VER = 1
+DEPS = libcdada
+###############################################################################
+# Top-level Rules +###############################################################################
+objects = $(DL_FILE)
+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
+$(DL_FILE)_MD5 = b829cfdc394d5acd6d32fb23de181a6b
+install : $(TARGET)
+check : $(patsubst %,$(DIR_CHK)/%,$(objects))
+download :$(patsubst %,$(DIR_DL)/%,$(objects))
+md5 : $(subst %,%_MD5,$(objects))
+dist: + @$(PAK)
+###############################################################################
+# Downloading, checking, md5sum +###############################################################################
+$(patsubst %,$(DIR_CHK)/%,$(objects)) : + @$(CHECK)
+$(patsubst %,$(DIR_DL)/%,$(objects)) : + @$(LOAD)
+$(subst %,%_MD5,$(objects)) : + @$(MD5)
+###############################################################################
+# Installation Details +###############################################################################
+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) + @$(PREBUILD) + @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE) + cd $(DIR_APP) && ./autogen.sh + cd $(DIR_APP) && ./configure \ + --prefix=/usr \ + --sysconfdir=/etc/pmacct \ + --enable-sqlite3 \ + --enable-l2 \ + --enable-jansson
+ # Add directory for plugins and pipes + -mkdir -v /var/spool/pmacct
+ # Add configuration folder and copy conf files to it + -mkdir -v /etc/pmacct + cp -vf $(DIR_CONF)/pmacct/*.conf /etc/pmacct
+ # Copy initscript to dir + cp -vf $(DIR_CONF)/pmacct/pmacct.init /etc/rc.d/init.d/pmacct + chmod 754 /etc/rc.d/init.d/pmacct + chown root:root /etc/rc.d/init.d/pmacct
+ cd $(DIR_APP) && make $(MAKETUNING) + cd $(DIR_APP) && make install
+ # Install backup definition + install -v -m 644 ${DIR_SRC}/config/backup/includes/pmacct \ + /var/ipfire/backup/addons/includes/pmacct + @rm -rf $(DIR_APP) + @$(POSTBUILD) diff --git a/make.sh b/make.sh index 43cccaca0..d42c9b1e2 100755 --- a/make.sh +++ b/make.sh @@ -1646,6 +1646,8 @@ buildipfire() { lfsmake2 ncdu lfsmake2 lshw lfsmake2 socat + lfsmake2 libcdada + lfsmake2 pmacct }
buildinstaller() { diff --git a/src/paks/pmacct/install.sh b/src/paks/pmacct/install.sh new file mode 100644 index 000000000..8178cc7bc --- /dev/null +++ b/src/paks/pmacct/install.sh @@ -0,0 +1,36 @@ +#!/bin/bash +############################################################################
+# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################
+# +. /opt/pakfire/lib/functions.sh
+extract_files +restore_backup ${NAME}
+# Add symlinks for runlevels +ln -s ../init.d/${NAME} /etc/rc.d/rc0.d/K85${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc3.d/S50${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc6.d/K85${NAME} +start_service --background ${NAME}
+# EOF diff --git a/src/paks/pmacct/uninstall.sh b/src/paks/pmacct/uninstall.sh new file mode 100644 index 000000000..51631e8c1 --- /dev/null +++ b/src/paks/pmacct/uninstall.sh @@ -0,0 +1,33 @@ +#!/bin/bash +############################################################################
+# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################
+# +. /opt/pakfire/lib/functions.sh
+stop_service ${NAME} +make_backup ${NAME} +remove_files
+# Delete symlinks +rm -rfv /etc/rc.d/rc?.d/???${NAME}
+# EOF diff --git a/src/paks/pmacct/update.sh b/src/paks/pmacct/update.sh new file mode 100644 index 000000000..89c40d0d7 --- /dev/null +++ b/src/paks/pmacct/update.sh @@ -0,0 +1,26 @@ +#!/bin/bash +############################################################################
+# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +############################################################################
+# +. /opt/pakfire/lib/functions.sh +./uninstall.sh
+./install.sh
2.30.1
Hi all,
Am Freitag, dem 19.03.2021 um 10:20 +0100 schrieb Adolf Belka:
Hi Michael,
On 18/03/2021 19:43, Adolf Belka wrote:
Hi Michael,
On 18/03/2021 19:25, Michael Tremer wrote:
Hey,
On 18 Mar 2021, at 13:34, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 18/03/2021 14:07, Michael Tremer wrote:
Hello, I will split this email into two parts: The technical part:
- The initscript isn’t based on our template and uses sleep
to check if starting a daemon has been successful. That should not be necessary and will potentially slow down the boot process.
I will have another look at the initscript and see if it can use the standard IPFire template.
Yes, it is not too different from your version:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/initscripts/system/tem...
There isn’t much documentation around it but there are loads of other scripts that will be helpful to figure out any convention or boilerplate code to copy.
- I am not very sure if the example configuration files are
helpful. Maybe someone can clear this up a little bit for me what is the rationale is behind them.
The idea was that the two simplest config example scripts were provided that give a user the ability to get data into memory or into a sqlite file. The actual conf file provided is a copy of the memory version. These give a basic starting point that people can then add additional info to be recorded etc.
Does anybody want to use anything else than SQLite in the context of IPFire?
I think we weren't sure and the memory and sqlite options were the easiest to get going and give real data. There are also mysql and postgresql options but I think we need to see what the interest is.
there would be needed the enable switch in configure to use pgsql or mysql which is not enabled in this LFS. The memory table is cheaper according to system resources and is able to feed also 3rd party solutions like e.g. RRD.
- There are loads of daemons being shipped and only one is
being used.
I looked through our conversations and it looks like these were left there because we weren't sure if they would be used or not.
pmacctd is the daemon being used by Jon. pmacct is the frontend to pmacctd when being used in memory mode.
For the others I looked in the pmacct website and found the following:-
I looked through our conversations and it looks like these were left there because we weren't sure if they would be used or not.
pmacctd is the daemon being used by Jon. pmacct is the frontend to pmacctd when being used in memory mode.
Exactly
For the others I looked in the pmacct website and found the following:-
'pmacctd' is the libpcap-based IPv4/IPv6 accounting daemon
'nfacctd' is the NetFlow (versions supported NetFlow v1, v5 and v9) and IPFIX accounting daemon
'sfacctd' is the sFlow v2/v4 and v5 accounting daemon
'uacctd' is the Linux NFLOG-based accounting daemon (historically, it was using ULOG, hence its name)
'pmtelemetryd' is the Streaming Telemetry collector, where "Streaming telemetry [ .. ] data can be used for analysis and troubleshooting purposes to maintain the health of the network. This is achieved by leveraging the capabilities of machine-to-machine communication. [ .. ]" (quoting Cisco IOS-XR Telemetry Configuration Guide at the time of this writing)
'pmbgpd' is the pmacct BGP collector and 'pmbmpd'is the pmacct BMP collector.
Pmacct can also handle BGP data via nfacct and the *bgp* binaries.
I don't know what most of that means but it looks to be quite different to what the addon was created for, so I would propose that I re-do the rootfiles and remove those daemons from the install. If in future there is demand for others we can add them back in with an update patch. I will do that together with the other changes already discussed.
As a beside info.
Best,
Erik
Regards,
Adolf.
I think Jon or Erik need to comment on this question.
- How much load does this generate? How large is the captured
data?
Jon needs to provide this feedback.
Then there is the other part:
On 8 Mar 2021, at 11:57, Adolf Belka (ipfire) adolf.belka@ipfire.org wrote:
From: Adolf Belka adolf.belka@ipfire.org
- What is it?
pmacct is a monitoring tool for network management tasks. Data collected can be used for analysis and troubleshooting purposes to maintain the health of the network. pmacct can collect, replicate and export network information. It can cache in memory tables, store persistently to SQLite3 and output to flat-files like CSV, formatted, and JSON.
Okay.
- Why is it needed?
To monitor data usage (IP-based or MAC-based data accounting) down to the client level. Net-Traffic will monitor traffic for the entire RED, GREEN, etc. networks, but it cannot pinpoint which client is using lots of data. Connections will take a snapshot but not show day by day sums. pmacct can help admins keep tabs on users that use too much data.
I agree with the goal of it. But how does this add-on achieve that?
Jon should comment on this as he had this use case and can give feedback how the data has helped him in his investigations.
The data is being written to a (usually binary) file which cannot be visualised in the web UI. Does this not want a CGI script that allows people to see what is happening?
We all thought that this would ultimately need a CGI script to set up and visualise the data. We believed it was worth getting this running in actual use as a CLI tool to start with and then get more feedback on the best items that would need to be captured in the CGI script. The tool has a lot of capabilities and I think it will not be possible to do everything that it can do in a WUI page. So some form of file or db offload will need to be available anyway but the items identified as being commonly required would be the ones to be developed into a CGI WUI page.
Sending manual SQL queries probably isn’t much fun :)
- What are the use cases?
An ISP may implement data caps and if the limit is over- run then you have to pay for every additional xxGB of data used. Typical charges can be around $10 per 50GB. With pmacct you can identify the high users and take action, hopefully before the limit is breached.
- This is being introduced as a command line only tool.
However, at a later date, if it is useful to enough additional users a WUI page could be developed as discussed in the development mailing list
https://lists.ipfire.org/pipermail/development/2021-January/009174.html
I have been thinking about this for a while now and would say that I can approve merging this as a CLI tool. But it would not really enhance IPFire’s functionality because it is only usable for people who spend a lot of time writing their own scripts or SQL queries to get information that can be used to make decisions.
We recognise that and had discussed the CGI WUI page as mentioned above.
Was this here on the list or elsewhere?
This was in a private conversation on the IPFire Community forum. We thought it best to have all our back and forth discussions between ourselves to start with. We were all finding our feet with it as well and we thought it best not to swamp the development list with our musings and questions to each other. If you want to see all the discussion history, then we can add you to the members of that conversation. If the addon gets accepted then any future discussions would be done on the Development mailing list to get everyones input.
Adding another alternative to vnstat will result in us having to maintain two things that roughly do the same - or where one’s features are a subset of the other one.
I think Jon or Erik need to comment on any comparison with vnstat.
I would like to avoid merging this and then see it abandoned because it is too complicated to use. We already have a lot of virtually unmaintained stuff around and I would really like to drop a lot of it, although I know that there are a few people actually using it.
Jon and Erik should comment on the complication of use or not. Jon especially has done quite a bit of work with it, installing the .ipfire package versions that I have created. I intend to also have a play around with it but for me it is more some playful fun as I just have a home network with only myself as the user and admin.
Thoughts? Best, -Michael
Co-authored-by: Jon Murphy Co-authored-by: Erik Kapfer ummeegge@ipfire.org
Signed-off-by: Adolf Belka adolf.belka@ipfire.org
config/backup/includes/pmacct | 2 + config/pmacct/pmacct.conf | 24 +++++ config/pmacct/pmacct.init | 57 +++++++++++ config/pmacct/pmacct_memory_example.conf | 24 +++++ config/pmacct/pmacct_sqlite3_example.conf | 25 +++++ config/rootfiles/packages/pmacct | 111 ++++++++++++++++++++++ lfs/pmacct | 107 +++++++++++++++++++++ make.sh | 2 + src/paks/pmacct/install.sh | 36 +++++++ src/paks/pmacct/uninstall.sh | 33 +++++++ src/paks/pmacct/update.sh | 26 +++++ 11 files changed, 447 insertions(+) create mode 100644 config/backup/includes/pmacct create mode 100644 config/pmacct/pmacct.conf create mode 100644 config/pmacct/pmacct.init create mode 100644 config/pmacct/pmacct_memory_example.conf create mode 100644 config/pmacct/pmacct_sqlite3_example.conf create mode 100644 config/rootfiles/packages/pmacct create mode 100644 lfs/pmacct create mode 100644 src/paks/pmacct/install.sh create mode 100644 src/paks/pmacct/uninstall.sh create mode 100644 src/paks/pmacct/update.sh
diff --git a/config/backup/includes/pmacct b/config/backup/includes/pmacct new file mode 100644 index 000000000..4359067ad --- /dev/null +++ b/config/backup/includes/pmacct @@ -0,0 +1,2 @@ +/etc/pmacct/ +/var/spool/pmacct/ diff --git a/config/pmacct/pmacct.conf b/config/pmacct/pmacct.conf new file mode 100644 index 000000000..3c1d47efa --- /dev/null +++ b/config/pmacct/pmacct.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip \ No newline at end of file diff --git a/config/pmacct/pmacct.init b/config/pmacct/pmacct.init new file mode 100644 index 000000000..31f9ce49b --- /dev/null +++ b/config/pmacct/pmacct.init @@ -0,0 +1,57 @@ +#!/bin/bash
+# Begin $rc_base/init.d/pmacct
+# Starts and stops pmacct daemon +# Date: 2021-02-25 12:25:30 (Thu, 25 Feb 2021)
+# Locations +CONF="/etc/pmacct/pmacct.conf" +CONTENT="/var/pmacct" +# Pmacctd binary +PMACCT="/usr/sbin/pmacctd" +# Pmacctd start options '-f' for the configuration file +OPT="-f ${CONF}" +DESC="pmacct daemon" +# Pmacctd syslog facilitiy +#SYSLOG="local1"
+. /etc/sysconfig/rc +. $rc_functions
+case "$1" in + start) + boot_mesg "Starting the ${DESC}... " + loadproc ${PMACCT} ${OPT} + sleep 2 + if P=$(pgrep -fl pmacctd); then + echo -e "\n${P}" + else + echo -e "\n${PMACCT} is not running... " + fi + ;;
+ stop) + boot_mesg "Stopping the ${DESC}... " + killproc ${PMACCT} + ;;
+ restart) + $0 stop + sleep 1 + $0 start + ;;
+ status) + statusproc ${PMACCT} + ;;
+ *) + echo "Usage: $0 {start|stop|restart|status}" + exit 1 + ;; +esac
+# End $rc_base/init.d/pmacct
diff --git a/config/pmacct/pmacct_memory_example.conf b/config/pmacct/pmacct_memory_example.conf new file mode 100644 index 000000000..0babdaaf7 --- /dev/null +++ b/config/pmacct/pmacct_memory_example.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin1" plugin configuration +! +plugins: memory[plugin1]
+plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000
+imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe
+aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip diff --git a/config/pmacct/pmacct_sqlite3_example.conf b/config/pmacct/pmacct_sqlite3_example.conf new file mode 100644 index 000000000..7fb996408 --- /dev/null +++ b/config/pmacct/pmacct_sqlite3_example.conf @@ -0,0 +1,25 @@ +! +! Pmacctd configuration file for IPFire environment +!
+syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0
+! +! "plugin2" plugin configuration +! +plugins: sqlite3[plugin2]
+sql_db[plugin2]: /var/spool/pmacct/pmacct_sqlitev1.db +sql_table_version[plugin2]: 1 +sql_history[plugin2]: 5m +sql_refresh_time[plugin2]: 300 +sql_history_roundoff[plugin2]: m +sql_dont_try_update: true
+aggregate[plugin2]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin2]: ip diff --git a/config/rootfiles/packages/pmacct b/config/rootfiles/packages/pmacct new file mode 100644 index 000000000..fed83ee67 --- /dev/null +++ b/config/rootfiles/packages/pmacct @@ -0,0 +1,111 @@ +etc/pmacct +etc/pmacct/pmacct.conf +etc/pmacct/pmacct_memory_example.conf +etc/pmacct/pmacct_sqlite3_example.conf +etc/rc.d/init.d/pmacct +usr/bin/pmacct +#usr/lib/pmacct +#usr/lib/pmacct/examples +#usr/lib/pmacct/examples/custom +#usr/lib/pmacct/examples/custom/libcustom.la +#usr/lib/pmacct/examples/lg +#usr/lib/pmacct/examples/lg/pmbgp +usr/sbin/nfacctd +usr/sbin/pmacctd +usr/sbin/pmbgpd +usr/sbin/pmbmpd +usr/sbin/pmtelemetryd +usr/sbin/sfacctd +usr/share/pmacct +#usr/share/pmacct/CONFIG-KEYS +#usr/share/pmacct/FAQS +#usr/share/pmacct/QUICKSTART +#usr/share/pmacct/UPGRADE +#usr/share/pmacct/docs +#usr/share/pmacct/docs/IPFIX +#usr/share/pmacct/docs/LOOKING_GLASS_FORMAT +#usr/share/pmacct/docs/MSGLOG_DUMP_FORMATS +#usr/share/pmacct/docs/SIGNALS +#usr/share/pmacct/docs/TRIGGER_VARS +#usr/share/pmacct/examples +usr/share/pmacct/examples/allow.lst.example +#usr/share/pmacct/examples/amqp +usr/share/pmacct/examples/amqp/amqp_receiver.py +#usr/share/pmacct/examples/avro +usr/share/pmacct/examples/avro/avro_file_decoder.py +usr/share/pmacct/examples/bgp_agent.map.example +usr/share/pmacct/examples/bgp_md5.lst.example +usr/share/pmacct/examples/bgp_xconnects.map.example +usr/share/pmacct/examples/flow_to_rd.map.example +#usr/share/pmacct/examples/kafka +usr/share/pmacct/examples/kafka/kafka_consumer.py +#usr/share/pmacct/examples/lg +usr/share/pmacct/examples/lg/pmbgp.py +usr/share/pmacct/examples/networks.lst.example +usr/share/pmacct/examples/nfacctd-sql.conf.example +usr/share/pmacct/examples/pcap_interfaces.map.example +usr/share/pmacct/examples/peers.map.example +usr/share/pmacct/examples/pmacctd-imt.conf.example +usr/share/pmacct/examples/pmacctd-multiple- plugins.conf.example +usr/share/pmacct/examples/pmacctd-sql.conf.example +usr/share/pmacct/examples/pmacctd-sqlite3.conf.example +usr/share/pmacct/examples/ports.lst.example +usr/share/pmacct/examples/pretag.map.example +usr/share/pmacct/examples/primitives.lst.example +usr/share/pmacct/examples/probe_netflow.conf.example +usr/share/pmacct/examples/probe_sflow.conf.example +usr/share/pmacct/examples/sampling.map.example +usr/share/pmacct/examples/tee_receivers.lst.example +#usr/share/pmacct/sql +#usr/share/pmacct/sql/README.64bit +#usr/share/pmacct/sql/README.GeoIP +#usr/share/pmacct/sql/README.IPv6 +#usr/share/pmacct/sql/README.cos +#usr/share/pmacct/sql/README.custom_primitives +#usr/share/pmacct/sql/README.etype +#usr/share/pmacct/sql/README.export_proto +#usr/share/pmacct/sql/README.iface +#usr/share/pmacct/sql/README.label +#usr/share/pmacct/sql/README.mask +#usr/share/pmacct/sql/README.mpls +#usr/share/pmacct/sql/README.mysql +#usr/share/pmacct/sql/README.nat +#usr/share/pmacct/sql/README.pgsql +#usr/share/pmacct/sql/README.sampling +usr/share/pmacct/sql/README.sqlite3 +#usr/share/pmacct/sql/README.tag2 +#usr/share/pmacct/sql/README.timestamp +#usr/share/pmacct/sql/README.tunnel +#usr/share/pmacct/sql/pmacct-create-db.pgsql +#usr/share/pmacct/sql/pmacct-create-db_bgp_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v2.mysql +#usr/share/pmacct/sql/pmacct-create-db_v3.mysql +#usr/share/pmacct/sql/pmacct-create-db_v4.mysql +#usr/share/pmacct/sql/pmacct-create-db_v5.mysql +#usr/share/pmacct/sql/pmacct-create-db_v6.mysql +#usr/share/pmacct/sql/pmacct-create-db_v7.mysql +#usr/share/pmacct/sql/pmacct-create-db_v8.mysql +#usr/share/pmacct/sql/pmacct-create-db_v9.mysql +#usr/share/pmacct/sql/pmacct-create-table_bgp_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_bgp_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v2.pgsql +usr/share/pmacct/sql/pmacct-create-table_v2.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v3.pgsql +usr/share/pmacct/sql/pmacct-create-table_v3.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v4.pgsql +usr/share/pmacct/sql/pmacct-create-table_v4.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v5.pgsql +usr/share/pmacct/sql/pmacct-create-table_v5.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v6.pgsql +usr/share/pmacct/sql/pmacct-create-table_v6.sqlite3 +usr/share/pmacct/sql/pmacct-create-table_v7.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v7_v8.pgsql +usr/share/pmacct/sql/pmacct-create-table_v8.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v9.pgsql +usr/share/pmacct/sql/pmacct-create-table_v9.sqlite3 +#usr/share/pmacct/sql/pmacct-grant-db.mysql +var/ipfire/backup/addons/includes/pmacct +var/spool/pmacct diff --git a/lfs/pmacct b/lfs/pmacct new file mode 100644 index 000000000..c6cc5270e --- /dev/null +++ b/lfs/pmacct @@ -0,0 +1,107 @@ +########################################################## #####################
+# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2019 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +########################################################## #####################
+########################################################## #####################
+# Definitions +########################################################## #####################
+include Config
+VER = 1.7.6
+THISAPP = pmacct-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP) +PROG = pmacct +PAK_VER = 1
+DEPS = libcdada
+########################################################## #####################
+# Top-level Rules +########################################################## #####################
+objects = $(DL_FILE)
+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
+$(DL_FILE)_MD5 = b829cfdc394d5acd6d32fb23de181a6b
+install : $(TARGET)
+check : $(patsubst %,$(DIR_CHK)/%,$(objects))
+download :$(patsubst %,$(DIR_DL)/%,$(objects))
+md5 : $(subst %,%_MD5,$(objects))
+dist: + @$(PAK)
+########################################################## #####################
+# Downloading, checking, md5sum +########################################################## #####################
+$(patsubst %,$(DIR_CHK)/%,$(objects)) : + @$(CHECK)
+$(patsubst %,$(DIR_DL)/%,$(objects)) : + @$(LOAD)
+$(subst %,%_MD5,$(objects)) : + @$(MD5)
+########################################################## #####################
+# Installation Details +########################################################## #####################
+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) + @$(PREBUILD) + @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE) + cd $(DIR_APP) && ./autogen.sh + cd $(DIR_APP) && ./configure \ + --prefix=/usr \ + --sysconfdir=/etc/pmacct \ + --enable-sqlite3 \ + --enable-l2 \ + --enable-jansson
+ # Add directory for plugins and pipes + -mkdir -v /var/spool/pmacct
+ # Add configuration folder and copy conf files to it + -mkdir -v /etc/pmacct + cp -vf $(DIR_CONF)/pmacct/*.conf /etc/pmacct
+ # Copy initscript to dir + cp -vf $(DIR_CONF)/pmacct/pmacct.init /etc/rc.d/init.d/pmacct + chmod 754 /etc/rc.d/init.d/pmacct + chown root:root /etc/rc.d/init.d/pmacct
+ cd $(DIR_APP) && make $(MAKETUNING) + cd $(DIR_APP) && make install
+ # Install backup definition + install -v -m 644 ${DIR_SRC}/config/backup/includes/pmacct \ + /var/ipfire/backup/addons/includes/pmacct + @rm -rf $(DIR_APP) + @$(POSTBUILD) diff --git a/make.sh b/make.sh index 43cccaca0..d42c9b1e2 100755 --- a/make.sh +++ b/make.sh @@ -1646,6 +1646,8 @@ buildipfire() { lfsmake2 ncdu lfsmake2 lshw lfsmake2 socat + lfsmake2 libcdada + lfsmake2 pmacct }
buildinstaller() { diff --git a/src/paks/pmacct/install.sh b/src/paks/pmacct/install.sh new file mode 100644 index 000000000..8178cc7bc --- /dev/null +++ b/src/paks/pmacct/install.sh @@ -0,0 +1,36 @@ +#!/bin/bash +########################################################## ##################
+# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +########################################################## ##################
+# +. /opt/pakfire/lib/functions.sh
+extract_files +restore_backup ${NAME}
+# Add symlinks for runlevels +ln -s ../init.d/${NAME} /etc/rc.d/rc0.d/K85${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc3.d/S50${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc6.d/K85${NAME} +start_service --background ${NAME}
+# EOF diff --git a/src/paks/pmacct/uninstall.sh b/src/paks/pmacct/uninstall.sh new file mode 100644 index 000000000..51631e8c1 --- /dev/null +++ b/src/paks/pmacct/uninstall.sh @@ -0,0 +1,33 @@ +#!/bin/bash +########################################################## ##################
+# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +########################################################## ##################
+# +. /opt/pakfire/lib/functions.sh
+stop_service ${NAME} +make_backup ${NAME} +remove_files
+# Delete symlinks +rm -rfv /etc/rc.d/rc?.d/???${NAME}
+# EOF diff --git a/src/paks/pmacct/update.sh b/src/paks/pmacct/update.sh new file mode 100644 index 000000000..89c40d0d7 --- /dev/null +++ b/src/paks/pmacct/update.sh @@ -0,0 +1,26 @@ +#!/bin/bash +########################################################## ##################
+# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team info@ipfire.org. # +# # +########################################################## ##################
+# +. /opt/pakfire/lib/functions.sh +./uninstall.sh
+./install.sh
2.30.1