From mboxrd@z Thu Jan 1 00:00:00 1970
From: Michael Tremer <michael.tremer@ipfire.org>
To: development@lists.ipfire.org
Subject: Re: [PATCH 2/4] pakfire.cgi: Implement JavaScript log message display
Date: Thu, 02 Dec 2021 17:58:25 +0000
Message-ID: <18A11A6D-A0AC-4F7A-B1FC-5CE52C1C1F37@ipfire.org>
In-Reply-To: <7dac08bc-a83f-a400-bb29-6f950d2b3c30@leo-andres.de>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============3138576484898399943=="
List-Id: <development.lists.ipfire.org>
--===============3138576484898399943==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Hello,
> On 2 Dec 2021, at 16:30, Leo Hofmann <hofmann(a)leo-andres.de> wrote:
>=20
> Hi,
>=20
> Am 02.12.2021 um 16:59 schrieb Michael Tremer:
>> Hello,
>>=20
>>> On 2 Dec 2021, at 15:39, Leo-Andres Hofmann <hofmann(a)leo-andres.de> wro=
te:
>>>=20
>>> Currently the page becomes unresponsive while Pakfire is busy.
>>> This patch implements a AJAX/JSON driven log output, to provide
>>> continuous information to the user while Pakfire is running.
>>>=20
>>> The output is updated 1x per second, if the load should be too high,
>>> the interval can be change by writing to "pakfire.refreshInterval".
>>>=20
>>> Signed-off-by: Leo-Andres Hofmann <hofmann(a)leo-andres.de>
>>> ---
>>> html/cgi-bin/pakfire.cgi | 153 ++++++++++++++++++----
>>> html/html/include/pakfire.js | 241 +++++++++++++++++++++++++++++++++++
>>> 2 files changed, 368 insertions(+), 26 deletions(-)
>>> create mode 100644 html/html/include/pakfire.js
>>>=20
>>> diff --git a/html/cgi-bin/pakfire.cgi b/html/cgi-bin/pakfire.cgi
>>> index 7957bc154..e5f5f7d6a 100644
>>> --- a/html/cgi-bin/pakfire.cgi
>>> +++ b/html/cgi-bin/pakfire.cgi
>>> @@ -36,8 +36,11 @@ my %color =3D ();
>>> my %pakfiresettings =3D ();
>>> my %mainsettings =3D ();
>>>=20
>>> -&Header::showhttpheaders();
>>> +# Load general settings
>>> +&General::readhash("${General::swroot}/main/settings", \%mainsettings);
>>> +&General::readhash("/srv/web/ipfire/html/themes/ipfire/include/colors.tx=
t", \%color);
>>>=20
>>> +# Get CGI request data
>>> $cgiparams{'ACTION'} =3D '';
>>> $cgiparams{'VALID'} =3D '';
>>>=20
>>> @@ -46,12 +49,102 @@ $cgiparams{'DELPAKS'} =3D '';
>>>=20
>>> &Header::getcgihash(\%cgiparams);
>>>=20
>>> -&General::readhash("${General::swroot}/main/settings", \%mainsettings);
>>> -&General::readhash("/srv/web/ipfire/html/themes/ipfire/include/colors.tx=
t", \%color);
>>> +### Process AJAX/JSON request ###
>>> +if($cgiparams{'ACTION'} eq 'json-getstatus') {
>>> + # Send HTTP headers
>>> + _start_json_output();
>>> +
>>> + # Collect Pakfire status and log messages
>>> + my %status =3D (
>>> + 'running' =3D> &_is_pakfire_busy() || "0",
>>> + 'running_since' =3D> &General::age("$Pakfire::lockfile") || "0s",
>>> + 'reboot' =3D> (-e "/var/run/need_reboot") || "0"
>>> + );
>>> + my @messages =3D `tac /var/log/messages | sed -n '/pakfire:/{p;/Pakfire=
.*started/q}'`;
>>> +
>>> + # Start JSON file
>>> + print "{\n";
>>> +
>>> + foreach my $key (keys %status) {
>>> + my $value =3D $status{$key};
>>> + print qq{\t"$key": "$value",\n};
>>> + }
>>> +
>>> + # Print sanitized messages in reverse order to undo previous "tac"
>>> + print qq{\t"messages": [\n};
>>> + for my $index (reverse (0 .. $#messages)) {
>>> + my $line =3D $messages[$index];
>>> + $line =3D~ s/[[:cntrl:]<>&\\]+//g;
>>> +
>>> + print qq{\t\t"$line"};
>>> + print ",\n" unless $index < 1;
>>> + }
>>> + print "\n\t]\n";
>> What is the reason to =E2=80=9Ctac=E2=80=9D the log file first and then re=
verse the order again?
>>=20
>> Is it just to limit the length of the JSON array?
>>=20
>> It might be faster to read the entire file, grep out what we need and then=
throw away most of the array. Or push a line to the end of the array and rem=
ove one from the beginning if it is longer than a certain threshold.
>=20
> I wanted to make sure that only the output of the current Pakfire run is sh=
own. Therefore, I use tac and sed to read the logfile backwards until the las=
t "Pakfire ... started!" header is reached.
> This works very well, but then of course the messages array is also in reve=
rse order.
Okay, that makes sense. Maybe we should start logging things into a separate =
file to make things easier?
/var/log/messages can become really large.
A C program is probably the fastest that we could ever have, so might be good=
enough as a solution for me.
-Michael
>=20
> All the ideas I had required some form of "reverse", or I had to load the e=
ntire file in Perl and check every line. I assumed that tac & sed would be mo=
re efficient than any Perl solution I could come up with. I'll try to time th=
is and report back!
>=20
> Leo
>=20
>>=20
>>> +
>>> + # Finalize JSON file & stop
>>> + print "}";
>>> + exit;
>>> +}
>>> +
>>> +### Start pakfire page ###
>>> +&Header::showhttpheaders();
>>> +
>>> +###--- HTML HEAD ---###
>>> +my $extraHead =3D <<END
>>> +<style>
>>> + /* Pakfire log viewer */
>>> + section#pflog-header {
>>> + width: 100%;
>>> + display: flex;
>>> + text-align: left;
>>> + align-items: center;
>>> + column-gap: 20px;
>>> + }
>>> + #pflog-header > div:last-child {
>>> + margin-left: auto;
>>> + margin-right: 20px;
>>> + }
>>> + #pflog-header span {
>>> + line-height: 1.3em;
>>> + }
>>> + #pflog-header span:empty::before {
>>> + content: "\\200b"; /* zero width space */
>>> + }
>>> +
>>> + pre#pflog-messages {
>>> + margin-top: 0.7em;
>>> + padding-top: 0.7em;
>>> + border-top: 0.5px solid $Header::bordercolour;
>>>=20
>>> -&Header::openpage($Lang::tr{'pakfire configuration'}, 1);
>>> + text-align: left;
>>> + min-height: 15em;
>>> + overflow-x: auto;
>>> + }
>>> +</style>
>>> +
>>> +<script src=3D"/include/pakfire.js"></script>
>>> +<script>
>>> + // Translations
>>> + pakfire.i18n.load({
>>> + 'working': '$Lang::tr{'pakfire working'}',
>>> + 'finished': 'Pakfire is finished! Please check the log output.',
>>> + 'since': '$Lang::tr{'since'} ', //(space is intentional)
>>> +
>>> + 'link_return': '<a href=3D"$ENV{'SCRIPT_NAME'}">Return to Pakfire</a>',
>>> + 'link_reboot': '<a href=3D"/cgi-bin/shutdown.cgi">$Lang::tr{'needreboo=
t'}</a>'
>>> + });
>>> +=09
>>> + // AJAX auto refresh interval
>>> + pakfire.refreshInterval =3D 1000;
>>> +</script>
>>> +END
>>> +;
>>> +###--- END HTML HEAD ---###
>>> +
>>> +&Header::openpage($Lang::tr{'pakfire configuration'}, 1, $extraHead);
>>> &Header::openbigbox('100%', 'left', '', $errormessage);
>>>=20
>>> +# Process Pakfire commands
>>> if (($cgiparams{'ACTION'} eq 'install') && (! &_is_pakfire_busy())) {
>>> my @pkgs =3D split(/\|/, $cgiparams{'INSPAKS'});
>>> if ("$cgiparams{'FORCE'}" eq "on") {
>>> @@ -170,29 +263,30 @@ if ($errormessage) {
>>> &Header::closebox();
>>> }
>>>=20
>>> -# Check if pakfire is already running.
>>> -if (&_is_pakfire_busy()) {
>>> - &Header::openbox( 'Waiting', 1, "<meta http-equiv=3D'refresh' content=
=3D'10;'>" );
>>> - print <<END;
>>> - <table>
>>> - <tr><td>
>>> - <img src=3D'/images/indicator.gif' alt=3D'$Lang::tr{'active'}' title=
=3D'$Lang::tr{'active'}' />
>>> - <td>
>>> - $Lang::tr{'pakfire working'}
>>> - <tr><td colspan=3D'2' align=3D'center'>
>>> - <form method=3D'post' action=3D'$ENV{'SCRIPT_NAME'}'>
>>> - <input type=3D'image' alt=3D'$Lang::tr{'reload'}' title=3D'$Lang::tr=
{'reload'}' src=3D'/images/view-refresh.png' />
>>> - </form>
>>> - <tr><td colspan=3D'2' align=3D'left'><code>
>>> -END
>>> - my @output =3D `grep pakfire /var/log/messages | tail -20`;
>>> - foreach (@output) {
>>> - print "$_<br>";
>>> - }
>>> - print <<END;
>>> - </code>
>>> - </table>
>>> +# Show log output while Pakfire is running
>>> +if(&_is_pakfire_busy()) {
>>> + &Header::openbox("100%", "center", "Pakfire");
>>> +
>>> + print <<END
>>> +<section id=3D"pflog-header">
>>> + <div><img src=3D"/images/indicator.gif" alt=3D"$Lang::tr{'active'}" tit=
le=3D"$Lang::tr{'pagerefresh'}"></div>
>>> + <div>
>>> + <span id=3D"pflog-status">$Lang::tr{'pakfire working'}</span><br>
>>> + <span id=3D"pflog-time"></span><br>
>>> + <span id=3D"pflog-action"></span>
>>> + </div>
>>> + <div><a href=3D"$ENV{'SCRIPT_NAME'}"><img src=3D"/images/view-refresh.p=
ng" alt=3D"$Lang::tr{'refresh'}" title=3D"$Lang::tr{'refresh'}"></a></div>
>>> +</section>
>>> +
>>> +<!-- Pakfire log messages -->
>>> +<pre id=3D"pflog-messages"></pre>
>>> +<script>
>>> + pakfire.running =3D true;
>>> +</script>
>>> +
>>> END
>>> +;
>>> +
>>> &Header::closebox();
>>> &Header::closebigbox();
>>> &Header::closepage();
>>> @@ -320,3 +414,10 @@ sub _is_pakfire_busy {
>>> # Test presence of PID or lockfile
>>> return (($pakfire_pid) || (-e "$Pakfire::lockfile"));
>>> }
>>> +
>>> +# Send HTTP headers
>>> +sub _start_json_output {
>>> + print "Cache-Control: no-cache, no-store\n";
>>> + print "Content-Type: application/json\n";
>>> + print "\n"; # End of HTTP headers
>>> +}
>>> diff --git a/html/html/include/pakfire.js b/html/html/include/pakfire.js
>>> new file mode 100644
>>> index 000000000..0950870e0
>>> --- /dev/null
>>> +++ b/html/html/include/pakfire.js
>>> @@ -0,0 +1,241 @@
>>> +/*######################################################################=
#######
>>> +# =
#
>>> +# IPFire.org - A linux based firewall =
#
>>> +# Copyright (C) 2007-2021 IPFire Team <info(a)ipfire.org> =
#
>>> +# =
#
>>> +# This program is free software: you can redistribute it and/or modify =
#
>>> +# it under the terms of the GNU General Public License as published by =
#
>>> +# the Free Software Foundation, either version 3 of the License, or =
#
>>> +# (at your option) any later version. =
#
>>> +# =
#
>>> +# This program is distributed in the hope that it will be useful, =
#
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of =
#
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the =
#
>>> +# GNU General Public License for more details. =
#
>>> +# =
#
>>> +# You should have received a copy of the GNU General Public License =
#
>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. =
#
>>> +# =
#
>>> +########################################################################=
#####*/
>>> +
>>> +"use strict";
>>> +
>>> +// Pakfire Javascript functions (requires jQuery)
>>> +class PakfireJS {
>>> + constructor() {
>>> + //--- Public properties ---
>>> + // Translation strings
>>> + this.i18n =3D new PakfireI18N();
>>> +
>>> + //--- Private properties ---
>>> + // Status flags (access outside constructor only with setter/getter)
>>> + this._states =3D Object.create(null);
>>> + this._states.running =3D false;
>>> + this._states.reboot =3D false;
>>> +
>>> + // Status refresh helper
>>> + this._autoRefresh =3D {
>>> + delay: 1000, //Delay between requests (default: 1s)
>>> + jsonAction: 'getstatus', //CGI POST action parameter
>>> + timeout: 5000, //XHR timeout (5s)
>>> +
>>> + delayTimer: null, //setTimeout reference
>>> + jqXHR: undefined, //jQuery.ajax promise reference
>>> + get runningDelay() { //Waiting for end of delay
>>> + return (this.delayTimer !=3D=3D null);
>>> + },
>>> + get runningXHR() { //Waiting for CGI response
>>> + return (this.jqXHR && (this.jqXHR.state() =3D=3D=3D 'pending'));
>>> + },
>>> + get isRunning() {
>>> + return (this.runningDelay || this.runningXHR);
>>> + }
>>> + };
>>> + }
>>> +
>>> + //### Public properties ###
>>> +
>>> + // Pakfire is running (true/false)
>>> + set running(state) {
>>> + if(this._states.running !=3D=3D state) {
>>> + this._states.running =3D state;
>>> + this._states_onChange('running');
>>> + }
>>> + }
>>> + get running() {
>>> + return this._states.running;
>>> + }
>>> +
>>> + // Reboot needed (true/false)
>>> + set reboot(state) {
>>> + if(this._states.reboot !=3D=3D state) {
>>> + this._states.reboot =3D state;
>>> + this._states_onChange('reboot');
>>> + }
>>> + }
>>> + get reboot() {
>>> + return this._states.reboot;
>>> + }
>>> +
>>> + // Status refresh interval in ms
>>> + set refreshInterval(delay) {
>>> + if(delay < 500) {
>>> + delay =3D 500; //enforce reasonable minimum
>>> + }
>>> + this._autoRefresh.delay =3D delay;
>>> + }
>>> + get refreshInterval() {
>>> + return this._autoRefresh.delay;
>>> + }
>>> +
>>> + // Document loaded (call once from jQuery.ready)
>>> + documentReady() {
>>> + // Status refresh late start
>>> + if(this.running && (! this._autoRefresh.isRunning)) {
>>> + this._autoRefresh_runNow();
>>> + }
>>> + }
>>> +
>>> + //### Private properties ###
>>> +
>>> + // Pakfire status change handler
>>> + // property: Affected status (running, reboot, ...)
>>> + _states_onChange(property) {
>>> + // Always update UI
>>> + if(this.running) {
>>> + $('#pflog-status').text(this.i18n.get('working'));
>>> + $('#pflog-action').empty();
>>> + } else {
>>> + $('#pflog-status').text(this.i18n.get('finished'));
>>> + if(this.reboot) { //Enable return or reboot links in UI
>>> + $('#pflog-action').html(this.i18n.get('link_reboot'));
>>> + } else {
>>> + $('#pflog-action').html(this.i18n.get('link_return'));
>>> + }
>>> + }
>>> +
>>> + // Start/stop status refresh if Pakfire started/stopped
>>> + if(property =3D=3D=3D 'running') {
>>> + if(this.running) {
>>> + this._autoRefresh_runNow();
>>> + } else {
>>> + this._autoRefresh_clearSchedule();
>>> + }
>>> + }
>>> + }
>>> +
>>> + //--- Status refresh scheduling functions ---
>>> +
>>> + // Immediately perform AJAX status refresh request
>>> + _autoRefresh_runNow() {
>>> + if(this._autoRefresh.runningXHR) {
>>> + return; // Don't send multiple requests
>>> + }
>>> + this._autoRefresh_clearSchedule(); // Stop scheduled refresh, will sen=
d immediately
>>> +
>>> + // Send AJAX request, attach listeners
>>> + this._autoRefresh.jqXHR =3D this._JSON_get(this._autoRefresh.jsonActio=
n, this._autoRefresh.timeout);
>>> + this._autoRefresh.jqXHR.done(function() { // Request succeeded
>>> + if(this.running) { // Keep refreshing while Pakfire is running
>>> + this._autoRefresh_scheduleRun();
>>> + }
>>> + });
>>> + this._autoRefresh.jqXHR.fail(function() { // Request failed
>>> + this._autoRefresh_scheduleRun(); // Try refreshing until valid status=
is received
>>> + });
>>> + }
>>> +
>>> + // Schedule next refresh
>>> + _autoRefresh_scheduleRun() {
>>> + if(this._autoRefresh.runningDelay || this._autoRefresh.runningXHR) {
>>> + return; // Refresh already scheduled or in progress
>>> + }
>>> + this._autoRefresh.delayTimer =3D window.setTimeout(function() {
>>> + this._autoRefresh.delayTimer =3D null;
>>> + this._autoRefresh_runNow();
>>> + }.bind(this), this._autoRefresh.delay);
>>> + }
>>> +
>>> + // Stop scheduled refresh (can still be refreshed up to 1x if XHR is al=
ready sent)
>>> + _autoRefresh_clearSchedule() {
>>> + if(this._autoRefresh.runningDelay) {
>>> + window.clearTimeout(this._autoRefresh.delayTimer);
>>> + this._autoRefresh.delayTimer =3D null;
>>> + }
>>> + }
>>> +
>>> + //--- JSON request & data handling ---
>>> +
>>> + // Load JSON data from Pakfire CGI, using a POST request
>>> + // action: POST paramter "json-[action]"
>>> + // maxTime: XHR timeout, 0 =3D no timeout
>>> + _JSON_get(action, maxTime =3D 0) {
>>> + return $.ajax({
>>> + url: '/cgi-bin/pakfire.cgi',
>>> + method: 'POST',
>>> + timeout: maxTime,
>>> + context: this,
>>> + data: {'ACTION': `json-${action}`},
>>> + dataType: 'json' //automatically check and convert result
>>> + })
>>> + .done(function(response) {
>>> + this._JSON_process(action, response);
>>> + });
>>> + }
>>> +
>>> + // Process successful response from Pakfire CGI
>>> + // action: POST paramter "json-[action]" used to send request
>>> + // data: JSON data object
>>> + _JSON_process(action, data) {
>>> + // Pakfire status refresh
>>> + if(action =3D=3D=3D this._autoRefresh.jsonAction) {
>>> + // Update status flags
>>> + this.running =3D (data['running'] !=3D '0');
>>> + this.reboot =3D (data['reboot'] !=3D '0');
>>> +
>>> + // Update timer display
>>> + if(this.running && data['running_since']) {
>>> + $('#pflog-time').text(this.i18n.get('since') + data['running_since']=
);
>>> + } else {
>>> + $('#pflog-time').empty();
>>> + }
>>> +
>>> + // Print log messages
>>> + let messages =3D "";
>>> + data['messages'].forEach(function(line) {
>>> + messages +=3D `${line}\n`;
>>> + });
>>> + $('#pflog-messages').text(messages);
>>> + }
>>> + }
>>> +}
>>> +
>>> +// Simple translation strings helper
>>> +// Format: {key: "translation"}
>>> +class PakfireI18N {
>>> + constructor() {
>>> + this._strings =3D Object.create(null); //Object without prototypes
>>> + }
>>> +
>>> + // Get translation
>>> + get(key) {
>>> + if(Object.prototype.hasOwnProperty.call(this._strings, key)) {
>>> + return this._strings[key];
>>> + }
>>> + return `(undefined string '${key}')`;
>>> + }
>>> +
>>> + // Load key/translation object
>>> + load(translations) {
>>> + if(translations instanceof Object) {
>>> + Object.assign(this._strings, translations);
>>> + }
>>> + }
>>> +}
>>> +
>>> +//### Initialize Pakfire ###
>>> +const pakfire =3D new PakfireJS();
>>> +
>>> +$(function() {
>>> + pakfire.documentReady();
>>> +});
>>> --=20
>>> 2.27.0.windows.1
--===============3138576484898399943==--