Xqm Beta Testing Phase 2, On Linux and Win32 platform ...

[hschneider]

You are invited to test XQM and XQM Agent on Win32 and Linux platform before they
are officially released.

What's new:
1. Message analyzing on the fly
That means there is an additional column in the grid view which contains XQM's
diagnosis tag for each message. E.g. QUOTA if the message bounced because the
user's mailbox was full, RELAY_DENIED if you were not allowed to relay etc.

Meanwhile its recognition library (parse.lib) was trained to tag 98% of all messages.

This allows you to quickly view a bulk of frozen messages without digging through
the SLOGs by hand. You can also sort and delete by that new attribute.
Please mail me SLOG files, that are not recogniced! This will help to improve
the recognition lib!

2. Scalable number of threads
The number of threads collecting the queue content in parallel can be scaled
from 1 to 25 for best possible performance also on slow machines.

3. Debug info
The Agent can be run in a debug mode on Win32 where it dumps a log to the console.
Just start it with paramter -debug from a command prompt.
On Linux it gives some useful informations during startup.

Download for Win32
XQM 1.27
http://xmail.marketmix.com/preview/XQMWin_Install.exe
XQM Agent 1.12
http://xmail.marketmix.com/preview/XQMAgen...Win_Install.exe

Download for Linux
XQM 1.27
http://xmail.marketmix.com/preview/xqmwin_linux.tgz
XQM Agent 1.13
http://xmail.marketmix.com/preview/xqmagent_linux.tgz

[chabral]

Well, I'm testing it with 8 threads and there are no performance improvements. The agent is using 65% of my server's memory and is slowing down the rest of the processes due to the increased use of swapping.
Here is the output (a significant part) of using "top":

76 processes: 66 sleeping, 10 running, 0 zombie, 0 stopped
CPU states: 69.1% user, 8.8% system, 0.0% nice, 21.9% idle
Mem: 130632K av, 128052K used, 2580K free, 0K shrd, 5240K buff
Swap: 265064K av, 126060K used, 139004K free 13500K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
16267 root 19 0 173M 82M 4496 R 10.0 65.0 0:14 xqmagent
16272 root 18 0 173M 82M 4496 R 9.8 65.0 0:17 xqmagent
16268 root 13 0 173M 82M 4496 R 8.4 65.0 0:12 xqmagent
16269 root 13 0 173M 82M 4496 R 7.9 65.0 0:12 xqmagent
16265 root 15 0 173M 82M 4496 R 7.5 65.0 0:15 xqmagent
16266 root 14 0 173M 82M 4496 R 7.5 65.0 0:11 xqmagent
16270 root 12 0 173M 82M 4496 R 7.1 65.0 0:12 xqmagent
16271 root 12 0 173M 82M 4496 R 6.9 65.0 0:14 xqmagent
16264 root 14 0 173M 82M 4496 R 5.0 65.0 0:13 xqmagent
4 root 8 0 0 0 0 SW 4.8 0.0 11:23 kswapd
16369 root 12 0 1076 1076 840 R 3.4 0.8 0:00 top
16132 root 2 0 173M 82M 4496 S 1.3 65.0 0:02 xqmagent
1 root 0 0 84 64 48 S 0.0 0.0 3:27 init

As you may remember, I've a Pentium Pro with 128MB of RAM.
I hope this information helps you,

Regards,

chabral

[hschneider]

Please testdrive with 4,2 and 1 thread.

[atomant]

I have running it with 1 thread and it is using 10% of CPU (my system is duron 650 with 256 megs RAM).

The Flusher doesn't work - No CTRL access ( xqmagent on my linux box).

[hschneider]

Are these 10% permanent or only when scanning the queue ?
CTRL Access is indeed out of order - your're an excellent tester :-)

[atomant]

These 10% is only when scanning, othervise is 0%. this is my top output:

CODE

199 processes: 197 sleeping, 2 running, 0 zombie, 0 stopped CPU states:  0.5% user,  4.7% system,  0.0% nice, 94.6% idle Mem:   255472K av,  203736K used,   51736K free,       0K shrd,   34396K buff Swap:  522104K av,   60732K used,  461372K free                  120348K cached  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND                                                                                 8142 root      16   0  1168 1168   840 R     5.3  0.4   0:02 top    1 root      15   0   476  440   428 S     0.0  0.1   0:04 init    2 root      15   0     0    0     0 SW    0.0  0.0   0:00 keventd    3 root      15   0     0    0     0 SW    0.0  0.0   1:06 kapmd    4 root      34  19     0    0     0 SWN   0.0  0.0   0:00 ksoftirqd_CPU0    5 root      15   0     0    0     0 SW    0.0  0.0   0:24 kswapd    6 root      15   0     0    0     0 SW    0.0  0.0   0:00 bdflush    7 root      15   0     0    0     0 SW    0.0  0.0   0:00 kupdated    8 root      25   0     0    0     0 SW    0.0  0.0   0:00 mdrecoveryd   12 root      15   0     0    0     0 SW    0.0  0.0   0:18 kjournald   77 root      15   0     0    0     0 SW    0.0  0.0   0:00 khubd  233 root      15   0     0    0     0 SW    0.0  0.0   0:00 kjournald  475 root      15   0     0    0     0 SW    0.0  0.0   0:00 eth0  542 root      15   0     0    0     0 SW    0.0  0.0   0:00 eth1  870 root      15   0   580  540   496 S     0.0  0.2   0:09 syslogd  874 root      15   0   428  412   376 S     0.0  0.1   0:04 klogd  896 root      15   0  3936  500   496 S     0.0  0.1   0:00 antivir  940 named     15   0  2564 2372  1720 S     0.0  0.9   0:23 named  956 root      15   0   840  776   736 S     0.0  0.3   0:00 xinetd  970 ntp       15   0  1908 1908  1864 S     0.0  0.7   0:00 ntpd  998 root      16   0  1100  936   936 S     0.0  0.3   0:00 safe_mysqld 1032 mysql     15   0  5264 1760  1740 S     0.0  0.6   0:00 mysqld 1047 root      15   0   644  596   564 S     0.0  0.2   0:00 crond 1060 root      16   0   896  752   752 S     0.0  0.2   0:00 squid 1063 squid     15   0  5428 3608  3160 R     0.0  1.4   2:31 squid 1064 squid     16   0   236  208   208 S     0.0  0.0   0:00 unlinkd 1080 root      15   0  5976 3988  1364 S     0.0  1.5   0:37 XMail 1111 daemon    15   0   516  496   460 S     0.0  0.1   0:00 atd 1122 nobody    16   0  1216  992   992 S     0.0  0.3   0:00 stunnel 1124 root      16   0   384  340   340 S     0.0  0.1   0:00 mingetty 1125 root      16   0   384  340   340 S     0.0  0.1   0:00 mingetty 1126 root      16   0   384  340   340 S     0.0  0.1   0:00 mingetty 1127 root      16   0   384  340   340 S     0.0  0.1   0:00 mingetty 1128 root      16   0   384  340   340 S     0.0  0.1   0:00 mingetty 1129 root      16   0   384  340   340 S     0.0  0.1   0:00 mingetty 1480 lp        15   0  1048  700   628 S     0.0  0.2   0:00 lpd 2115 root      15   0  1316  828   788 S     0.0  0.3   0:03 smbd 2119 root      15   0  1396  960   856 S     0.0  0.3   0:00 nmbd 32453 root      15   0  3696 2572  2540 S     0.0  1.0   0:39 httpd 6461 root      15   0  3680 3680  1556 S     0.0  1.4   0:00 xqmagent 8066 root      15   0   592  592   500 S     0.0  0.2   0:00 in.telnetd 8067 root      15   0  1356 1356   972 S     0.0  0.5   0:00 login 8068 sasa      15   0  1400 1400  1120 S     0.0  0.5   0:00 bash 8112 root      15   0  1008 1008   832 S     0.0  0.3   0:00 su

[hschneider]

Thats OK for browsing over 4.000 directories in the shortest possible time.

Chabral, I think reducing to 1 or 2 threads for the cost of time will be the best solution. Your machine seems to be very low on resources and starts to swap mem easily. Maybe you also should reduce the number of threads spawned by the Xmal process.

[hschneider]

XQM Agent 1.13 / Linux
http://xmail.marketmix.com/preview/xqmagent_linux.tgz

Flush() works again.

Update parse.lib attached (goes to your XQM folder).



Attached File ( Number of downloads: 166 )
 parse.lib

[atomant]

Yep, Flush() works.

[chabral]

Using XQM Agent 1.13 / Linux with 1 thread configured, I see 2 threads running and eating
around 50% of CPU and 50% of memory. There are also another 2 threads sleeping most of the time.
It took 15 minutes to XQM to show me the queue the first time, and 8 minutes the second time. There are still things to be done to improve performance of the Agent. Its eating more resources than the mail server (wich also uses the queue) and all other deamons thogheter.
This is the output of "top" using "memory usage" order.

69 processes: 66 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 93.7% user, 6.2% system, 0.0% nice, 0.0% idle
Mem: 130632K av, 128364K used, 2268K free, 0K shrd, 5240K buff
Swap: 265064K av, 82192K used, 182872K free 7692K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
25168 root 0 0 120M 70M 652 S 1.3 55.4 0:06 xqmagent
25237 root 0 0 120M 70M 652 S 0.0 55.4 0:00 xqmagent
25238 root 10 0 120M 70M 652 R 45.6 55.4 4:03 xqmagent
25239 root 10 0 120M 70M 652 R 45.0 55.4 4:17 xqmagent

And this is when the agent is not interacting with the GUI.

67 processes: 65 sleeping, 1 running, 1 zombie, 0 stopped
CPU states: 0.6% user, 1.8% system, 0.0% nice, 97.5% idle
Mem: 130632K av, 128388K used, 2244K free, 0K shrd, 7172K buff
Swap: 265064K av, 88276K used, 176788K free 23388K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
25168 root 0 0 109M 50M 660 S 0.0 39.5 1:17 xqmagent
25237 root 0 0 109M 50M 660 S 0.0 39.5 0:00 xqmagent

Is there another test I can do?
Regards,

chabral

[hschneider]

I managed to testrun the Agent on a
Athlon 550 MHz, 128 MB RAM, SUSE Linux 8.0, Kernel 2.4, KDE 3.0, ReiserFS

CODE

2 processes: 2 sleeping, 0 running, 0 zombie, 0 stopped   CPU states:  8.0% user, 11.7% system,  0.1% nice, 80.1% idle   Mem:   126384K av,  122104K used,    4280K free,       0K shrd,   19496K buff   Swap:  124904K av,    2040K used,  122864K free                   34932K cached       PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND  [0m 2187 root      20   0  4108 4108  1004 S     0.0  3.2   0:00 xqmagent   2231 root      20   0  4108 4108  1004 S     0.0  3.2   0:00 xqmagent

This is 0% CPU and 3.2% of mem when the Agent is idle. This is a 10th of the usage on your system. This is very strange since the mem usage has nothing to do with the CPU type.

Querying the queue with XQM on the same machine takes 16 seconds the 1st time and 8 seconds with 25 threads. There were 202 messages in the Q. This is a huge difference, which cannot be causes by the faster CPU and the different filesystem alone ...

I'm very interested in the perfomance on other systems .. so if anyone reads this, please mail a top dump!

>There are still things to be done to improve performance of the Agent ..
There is not very much to optimize in its code. The problem here is, that Python is a bytecode interpreter like Java. It's code is flexible but not as performant as pure C-Code. This doesn't matter on fast machines, but seems to matter in your environment ...

[Bhozar]

when quirying XQM running on the server uses 90% of CPU and always used 16mb of memory. This is on a dual PII 450mhz system with 1GB RAM Running windows 2000 Adv server and NTFS.


Here is a SLOG file that doesn show an error message:

[PeekTime] 1051226057 : Fri, 25 Apr 2003 00:14:17 +0100
<<
ErrCode = -232
ErrString = Error connecting to remote address
SMAIL SMTP-Send MX = "ywing.aoltw.net." SMTP = "networks.co.uk" From = "postmaster@networks.co.uk" To = "oqf3w0sr97@netscape.com" Failed !
SMTP-Error = "417 Temporary delivery error"
SMTP-Server = "ywing.aoltw.net."
>>
<<
ErrCode = -81
ErrString = [MAIL FROM:] not permitted by remote SMTP server
ErrInfo = 550 5.7.1 Access denied. IP name lookup failed [62.189.105.1]
SMAIL SMTP-Send MX = "awing.aoltw.net." SMTP = "networks.co.uk" From = "postmaster@networks.co.uk" To = "oqf3w0sr97@netscape.com" Failed !
SMTP-Error = "550 5.7.1 Access denied. IP name lookup failed [62.189.105.1]"
SMTP-Server = "awing.aoltw.net."
>>

[hschneider]

>when quirying XQM running on the server uses 90% of CPU and always used 16mb ...

Thanks for the info.
Querying is desinged to gather as much informations as possible in the shortest time frame. That means it allocates as much CPU bandwidth as possible. That's the reason why it takes snapshot and doesn't do a cyclic realtime query.

parse.lib update attached.

Attached File ( Number of downloads: 163 )
 parse.lib

[Bhozar]

New parse.lib fixed it.

[chabral]

Hi,
Which is the purpose of parse.lib?

Regards,

chabral