tag:blogger.com,1999:blog-49419228008812433462024-03-13T17:24:20.100-04:00Blog.PlatinumFactor.ComUnknownnoreply@blogger.comBlogger34125tag:blogger.com,1999:blog-4941922800881243346.post-33202120958922145692016-12-31T14:08:00.001-05:002016-12-31T15:09:03.937-05:00Zabbix with Galera/Percona Hourly Spikes. SOLVED!The relief I have in solving this problem is indescribable. Utterly indescribable!<br />
This had plagued us for months, and in such issues everyone in the team goes on the edge.<br />
<br />
<b>First: Which SPIKES do I speak of?</b><br />
Open Zabbix server's "Zabbix internal process busy %" graph.<br />
For us, this graph had crazy spikes every hour when the slowdown happened.<br />
<br />
<b>Why are the SPIKES caused?</b><br />
This is Zabbix's process which converts history to trend data every hour.<br />
<br />
<b>Our Setup</b><br />
Zabbix 2.4<br />
Percona (with MySQL) 5.6 DB - Geo-Distributed/WAN Cluster across 2 sites<br />
Current DB size: around 200 GB<br />
<br />
<b>Enter the SPIKES</b><br />
Every hour, on the hour, we get a serious slowdown. Zabbix UI freezes. Graphs start going gray.<br />
We looked at the resources: IO was high which CAUSED CPU to be high. Everything else looked fine. We moved one DB node to SSD storage. Things calmed down with resources BUT spike was still present.<br />
<br />
<b>Zabbix server logs show "duplicate entry" errors</b><br />
We looked at Zabbix logs and saw some "Duplicate Entry" errors.<br />
When I researched this, it indicated database corruption, which, in our case, was due to two Zabbix server processes running connected to same DB. Someone had accidentally started the 2nd (failover) server. We stopped the 2nd server and cleaned up the DB; dropped all history and trend data.<br />
<br />
Spikes went away, but we were not sure whether it was due to DB size or really it was the corruption. <br />
<br />
We waited and let the DB grow, and lo and behold, Spikes came back!<br />
<br />
Now we had at least eliminated the DB corruption issue.<br />
<br />
We started by tweaking Zabbix knobs first, nothing worked.<br />
<br />
<b>Replication Test</b><br />
Finally, we started looking at the Galera replication. The way we determined this was "replication" related was by a simple test: shut down all DB nodes except for the one Zabbix connects to.<br />
Once we did this, NO SPIKES!<br />
Test 2: Bring up a DB node in same Site - NO SPIKES!<br />
Test 3: Bring up a DB node in remote Site - SPIKES!<br />
<br />
<b>Results</b><br />
Galera replication is not handling our intra-site network too well.<br />
Let's start looking at replication and flow control options.<br />
<br />
Let's revisit Philip's presentation<br />
<a href="https://www.percona.com/live/europe-amsterdam-2015/sessions/using-galera-cluster-power-geo-distributed-applications-wan">https://www.percona.com/live/europe-amsterdam-2015/sessions/using-galera-cluster-power-geo-distributed-applications-wan</a><br />
<br />
All of this was already set.<br />
Further research yields<br />
<a href="https://www.percona.com/blog/2013/05/02/galera-flow-control-in-percona-xtradb-cluster-for-mysql/">https://www.percona.com/blog/2013/05/02/galera-flow-control-in-percona-xtradb-cluster-for-mysql/</a><br />
<br />
<pre style="background: #0c1021; color: #f8f8f8;">gcs.fc_limit=<span style="color: #d8fa3c;">500</span>; gcs.fc_master_slave=YES; gcs.fc_factor=<span style="color: #d8fa3c;">1.0</span>
</pre>
<br />
We set the nodes to use these FC settings. No help. SUPER FRUSTRATION!<br />
Ask in Galera forums, no help (or they are too busy or on holidays!)<br />
<br />
So back to reading on Galera parameters.<br />
I do a bunch of reading and find these. This indicates how many transactions are waiting to be processed in the queue. BINGO! <br />
<br />
<pre style="background: #0c1021; color: #f8f8f8;">mysql<span style="color: #fbde2d;">></span> SHOW GLOBAL STATUS <span style="color: #fbde2d;">LIKE</span> <span style="color: #61ce3c;">'wsrep_local_recv%'</span>;
<span style="color: #fbde2d;">+</span><span style="color: #aeaeae;">----------------------------+------------+</span>
| Variable_name | Value |
<span style="color: #fbde2d;">+</span><span style="color: #aeaeae;">----------------------------+------------+</span>
| wsrep_local_recv_queue | <span style="color: #d8fa3c;">1721</span> |
| wsrep_local_recv_queue_max | <span style="color: #d8fa3c;">1721</span> |
| wsrep_local_recv_queue_min | <span style="color: #d8fa3c;">0</span> |
| wsrep_local_recv_queue_avg | <span style="color: #d8fa3c;">169</span>.<span style="color: #d8fa3c;">347046</span> |
<span style="color: #fbde2d;">+</span><span style="color: #aeaeae;">----------------------------+------------+</span>
<span style="color: #d8fa3c;">4</span> rows <span style="color: #fbde2d;">in</span> <span style="color: #fbde2d;">set</span> (<span style="color: #d8fa3c;">0</span>.<span style="color: #d8fa3c;">00</span> sec)
</pre>
<br />
Our fc_limit value was too low! We were sometimes passing 3000 transactions!<br />
So obviously the next thing I did was to set fc_limit to a number higher than this, and DONE!<br />
<br />
To check how node has caught up, the wsrep_local_recv_queue should fall back to 0.<br />
In our case, that happens within few seconds.<br />
<br />
This was a great NEW YEAR'S PRESENT for me.<br />
And if any of you are using Percona/Galera with any application this information might help!<br />
<br />Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-4941922800881243346.post-24154241554249089032016-10-12T15:33:00.002-04:002016-10-12T15:33:27.060-04:00Pacemaker - pcs cluster auth does not work on CentOS 6.x<br />
Recently we were trying to fix an issue with our pacemaker/cman cluster on CentOS 6.7.<br />
Regardless of everything we tried, pcs cluster auth was not working.<br />
<br />
Started digging and found PAM blocking auth in /var/log/secure<br />
Upon investigation started looking at this file<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">/etc/pam.d/password-auth</span><br />
<br />
And commented out this line<br />
<span style="font-family: "Courier New",Courier,monospace;">auth required pam_listfile.so onerr=fail item=group sense=allow file=/etc/login.group.allowed</span><br />
<br />
This fixed the problem but since our PAM configs are pushed via puppet, they were being overwritten during puppet run so I kept looking and found this<br />
<br />
# in this file add <span style="font-family: "Courier New",Courier,monospace;">haclient</span><br /><span style="font-family: "Courier New",Courier,monospace;">vim /etc/login.group.allowed</span><br />
<br />
Check auth like this<br />
<span style="font-family: "Courier New",Courier,monospace;">pcs cluster auth nodeA nodeB -u hacluster </span><br />
<br />
Assumption is you already have a password set for hacluster user.<br />
If not then set it from root as:<br />
<span style="font-family: "Courier New",Courier,monospace;">passwd hacluster</span><br />
Then restart pcsd serviceUnknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-68317154733165749412016-05-22T13:54:00.003-04:002016-05-22T13:54:32.144-04:00Zabbix proxy force configuration update<br />
From Zabbix proxy shell run:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">zabbix_proxy
-R config_cache_reload</span></blockquote>
<br />
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4941922800881243346.post-17387697039846436462016-05-22T13:48:00.003-04:002016-05-22T13:49:15.480-04:00WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor<br />
The reason as we found for this error was one of the following<br />
<br />
<br />
<blockquote class="tr_bq">
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">Improper permissions
on /var/lib/mysql</span></div>
<div style="font-size: 11pt; margin: 0in;">
</div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">Sstuser account
doesn't have proper permissions</span></div>
</blockquote>
<br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-7230198160403997312016-05-22T13:45:00.004-04:002016-05-22T13:46:25.734-04:00Zabbix history and trends cleanupShut down the Zabbix server and frontend connections to the DB<br />
<br />
If using MySQL, make
sure auto_recalc is 1 otherwise you will have to analyze all these tables. The default value is 1.<br />
<br />
<br />
<blockquote class="tr_bq">
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">CREATE TABLE
history_new LIKE history;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">CREATE TABLE
history_log_new LIKE history_log;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">CREATE TABLE
history_str_new LIKE history_str;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">CREATE TABLE
history_text_new LIKE history_text;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">CREATE TABLE
history_uint_new LIKE history_uint;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">CREATE TABLE
trends_new LIKE trends;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">CREATE TABLE
trends_uint_new LIKE trends_uint;</span></div>
<div style="font-size: 11pt; margin: 0in;">
</div>
<div style="font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE trends
RENAME trends_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
trends_new RENAME trends;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
trends_uint RENAME trends_uint_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
trends_uint_new RENAME trends_uint;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE history
RENAME history_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
history_new RENAME history;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
history_log RENAME history_log_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
history_log_new RENAME history_log;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
history_str RENAME history_str_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
history_str_new RENAME history_str;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
history_text RENAME history_text_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
history_text_new RENAME history_text;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
history_uint RENAME history_uint_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE
history_uint_new RENAME history_uint;</span></div>
<div style="font-size: 11pt; margin: 0in;">
</div>
<div style="font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">DROP TABLE
trends_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">DROP TABLE
trends_uint_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">DROP TABLE
history_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">DROP TABLE
history_log_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">DROP TABLE
history_str_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">DROP TABLE
history_text_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">DROP TABLE
history_uint_old;</span></div>
<div style="font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-size: 11pt; margin: 0in;">
</div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: "courier new" , "courier" , monospace;">delete from events;</span></div>
</blockquote>
<br />
That's all. Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-77442143886289976412016-05-22T13:39:00.004-04:002016-05-22T13:42:45.755-04:00Duplicate Entry error in Zabbix server logs<br />
We were seeing the following error<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;">22199:201<span style="font-family: "courier new" , "courier" , monospace;">5</span>0<span style="font-family: "courier new" , "courier" , monospace;">6</span>13:133805.639
[Z3005] query failed: [1062] Duplicate entry '1743313' for key 'PRIMARY'
[insert into events (eventid,source,object,objectid,clock,ns,value) values
(1743313,3,0,55456,1460569085,540384532,0);</span></span></blockquote>
<br />
Main reason for this error is that two Zabbix servers were connected to the same DB server. We thought this might work in a HA scenario but it doesn't even though the second Zabbix server is doing nothing.<br />
<br />
You might also have to run this query<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">delete
from events;</span></blockquote>
<br />
Be careful as it would delete all events.<br />
We did not care about this in a load test so we were OK with it. Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-23242150201650237932015-08-13T19:48:00.000-04:002015-08-13T19:49:52.701-04:00ERROR listener failed: zbx_tcp_listen() fatal error: unable to serve on any address [[-]:10051]<span style="font-family: Verdana, sans-serif;">Load testing scenario and similar error to the post below, but different reason.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Zabbix
runs as part of Pacemaker cluster.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">There were 2 reasons:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">1. mySQL did not have enough connections allowed</span><br />
<span style="font-family: Verdana, sans-serif;">Set this parameter in my.cnf: max_connections
= 512</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">2. Zabbix service controlled by Pacemaker. This requires maintenance mode. I saw articles on how to do it with crm command but this is obsolete. So here is how with pcs</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Courier New, Courier, monospace;">pcs property set
maintenance-mode=true</span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-size: 11pt;"><span style="font-family: Courier New, Courier, monospace;">pcs property set
maintenance-mode=false</span></span></div>
<div style="font-size: 11pt; margin: 0in;">
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: Verdana, sans-serif;">Once it is in maintenance mode, it can controlled independently of Pacemaker.</span></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-66095327063590674502015-07-30T13:56:00.003-04:002015-07-30T13:59:04.316-04:00mysql-proxy not running via puppet service resource type<br />
Had an issue with service resource for mysql-proxy.<br />
When executing the manifest, the service would not start. If I try manually, the service runs. If I try with exec resource, the service runs.<br />
<br />
Seems the init script is not lsb compliant<br />
<a href="https://docs.puppetlabs.com/references/latest/type.html#service-attribute-status">https://docs.puppetlabs.com/references/latest/type.html#service-attribute-status</a><br />
<br />
<br />
<b>Modify as such to get it working</b><br />
<span style="font-family: Courier New, Courier, monospace;">service { 'mysql-proxy':</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ensure => "running",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> status => 'ps afx | grep -i mysql-proxy | grep -v grep',</span><br />
<span style="font-family: Courier New, Courier, monospace;"> hasstatus => "no",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> hasrestart => "yes",</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-14775277414890792732015-07-15T20:07:00.004-04:002015-07-22T11:30:19.424-04:00zabbix-server does not start - zbx_tcp_listen() fatal error<span style="font-family: Verdana, sans-serif;">Zabbix 2.4.x</span><br />
<span style="font-family: Verdana, sans-serif;">CentOS 6.6</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Although I have rarely seen something like this but my deployment of Zabbix via Puppet caused a very strange issue where Zabbix service would not start</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Error in logs</span><br />
<span style="font-family: Verdana, sans-serif;">listener failed: zbx_tcp_listen() fatal error: unable to serve on any address [[-]:10051]</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Service status and when trying to start</span><br />
<span style="font-family: Verdana, sans-serif;">[root@abc-zabserver-b zabbix]# service zabbix-server status</span><br />
<span style="font-family: Verdana, sans-serif;">zabbix_server is stopped</span><br />
<span style="font-family: Verdana, sans-serif;">[root@abc-zabserver-b zabbix]# service zabbix-server start</span><br />
<span style="font-family: Verdana, sans-serif;">Starting Zabbix server: [ OK ]</span><br />
<span style="font-family: Verdana, sans-serif;">[root@abc-zabserver-b zabbix]# service zabbix-server status</span><br />
<span style="font-family: Verdana, sans-serif;">zabbix_server is stopped</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Process is running (sometimes shows multiple processes running)</span><br />
<span style="font-family: Verdana, sans-serif;">[root@abc-zabserver-b zabbix]# ps afx | grep -i zabbix</span><br />
<span style="font-family: Verdana, sans-serif;"> 3852 pts/1 S+ 0:00 | \_ grep -i zabbix</span><br />
<span style="font-family: Verdana, sans-serif;"> 2150 ? S 0:00 zabbix_server -c /etc/zabbix/zabbix_server.conf</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">But service is still stopped</span><br />
<span style="font-family: Verdana, sans-serif;">[root@abc-zabserver-b zabbix]# service zabbix-server status</span><br />
<span style="font-family: Verdana, sans-serif;">zabbix_server is stopped</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">If I kill the process(es) then Zabbix service comes up fine.</span><br />
<span style="font-family: Verdana, sans-serif;">Cause</span><br />
<span style="font-family: Verdana, sans-serif;">When I was deploying Zabbix with Puppet I was using</span><br />
<span style="font-family: Verdana, sans-serif;">ensure => installed</span><br />
<span style="font-family: Verdana, sans-serif;">instead of</span><br />
<span style="font-family: Verdana, sans-serif;">ensure => '2.4.1.-5.el6' (or some other specific version)</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Basically my config file for Zabbix was still and older version and it didn't play too well with the new Zabbix that became available through the repos.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">I ended up refreshing the config file.</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-55256084627448074182015-07-13T19:57:00.003-04:002015-07-13T20:01:01.571-04:00Decrease timeout for Zabbix OK blinker<span style="font-family: Verdana, sans-serif;"><b>version: Zabbix Server 2.4</b></span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The default time for OK and status change trigger is 30 minutes which means the OK keeps blinking on the screen for that long. </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">There are two ways to change this:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">Go to Administration > General > Trigger displaying options (drop down on right)<br />
Change values of following as desired</span><br />
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">Display OK triggers for</span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">On status change triggers blink for</span></li>
</ul>
<span style="font-family: Verdana, sans-serif;"><br />The other way is to directly change it in DB (MySQL in this case):</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">mysql
-u zabbix -p <PASSWORD> -e 'UPDATE config SET `ok_period`=60,
`blink_period`=60' zabbixserverdb</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">This is more helpful for automation.</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-17261134478660036082015-07-09T03:25:00.001-04:002015-07-10T13:06:34.089-04:00MySQL server has gone away - Galera with HAProxy<br />
Whew! Been a long time. I've been working on some interesting projects and will update with more info soon. Recently, I created a MySQL Galera cluster with HAProxy load balancing.<br />
<br />
<b>I saw these errors</b><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">ERROR 2006 (HY000): MySQL server has gone away<br />No connection. Trying to reconnect...</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">[Z3005] query failed: [2006] MySQL server has gone away [select m.maintenanceid,m.maintenance_type,m.active_since,tp.timeperiod_type,tp.every,tp.month,tp.dayofweek,tp.day,tp.start_time,tp.period,tp.start_date from maintenances m,maintenances_windows mw,timeperiods tp where m.maintenanceid=mw.maintenanceid and mw.timeperiodid=tp.timeperiodid and m.active_since<=1436204460 and m.active_till>1436204460]</span></blockquote>
<br />
<b>Tweak the following</b><br />
<br />
On Galera nodes, add to /etc/my.cnf<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">wait_timeout = 28000<br />max_allowed_packet = 64M</span></blockquote>
<br />
In HAProxy config /etc/haproxy/haproxy.cfg increase timeout values<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">timeout connect 10s<br />timeout client 2m<br />timeout server 2m</span></blockquote>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-17515646230859171572012-10-21T20:13:00.002-04:002012-10-21T20:23:18.265-04:00OpenLDAP + TLS on CentOS 6Just what the subject says. Here are the guides that helped me:<br />
<a href="http://www.server-world.info/en/note?os=CentOS_6&p=ldap"><b>http://www.server-world.info/en/note?os=CentOS_6&p=ldap</b></a><br />
<b><a href="http://www.computerglitch.net/bin/texts/CentOS6_LDAP.php">http://www.computerglitch.net/bin/texts/CentOS6_LDAP.php</a></b><br />
<br />
If you want to include SAMBA:<br />
<a href="http://shirker.blog.com/2012/01/20/ldap-samba-phpldapadmin-on-centos-6/"><b>http://shirker.blog.com/2012/01/20/ldap-samba-phpldapadmin-on-centos-6/</b></a><br />
<br />
For PHPLDAPAdmin:<br />
<a href="http://www.server-world.info/en/note?os=CentOS_6&p=ldap&f=4"><b>http://www.server-world.info/en/note?os=CentOS_6&p=ldap&f=4</b></a><br />
<br />
I installed OpenLDAP for masochistic reasons. If you are doing an actual production deployment then seriously consider the 389 Directory Server in its place.Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4941922800881243346.post-66930233619017369332012-10-13T12:32:00.003-04:002012-10-13T12:35:32.283-04:00Windows Disk Space - HibernateI recently added more RAM to my system and my OS SSD disk which is 60 GB completely filled up.<br />
I was looking for some answers on Google and thought it might have something to do with the swap/pagefile etc. However, as it turns out this is an issue related to hibernation.<br />
<br />
Disable hibernate as such to resolve the issue:<br />
<a href="http://www.sevenforums.com/tutorials/819-hibernate-enable-disable.html"><b>http://www.sevenforums.com/tutorials/819-hibernate-enable-disable.html</b></a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-49712663191754658502012-03-04T15:39:00.001-05:002014-09-29T22:55:47.110-04:00Puppet vs Spacewalk (Satellite)<div>
<br />
We are evaluating configuration and package management tools and it's between the two mentioned applications. Puppet sounds great but it uses Ruby DSL and our team is currently implementing Zenoss (Python) and migrating a lot of internal scripts (from Perl to Python), so this will be interesting.</div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4941922800881243346.post-68175420137067743452011-08-12T12:24:00.000-04:002012-10-21T20:24:01.708-04:00Quote from /bin/fortune<b><i>If you make people think they're thinking, they'll love you, but if you really make them think, they'll hate you. - /usr/bin/fortune</i></b>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-58961335017144508702011-08-08T13:20:00.001-04:002011-08-08T13:21:52.911-04:00Changing vi / vim color scheme in CentOS 6I just did a minimal desktop install of CentOS 6 and did not like the default vi / vim color scheme.<br />
Here is where the color schemes are located:<br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b>/usr/share/vim/vim72/colors</b></span><br />
<br />
In the above example, <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>vim72</b></span> is the version of vim, yours could be different.<br />
First you have to copy <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>/etc/vimrc</b></span> to your home directory and rename to .vimrc<br />
Now edit .vimrc that is in your home directory and add the following,<br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b>:color <yourcolorscheme></b></span><br />
<br />
So instead of <yourcolorscheme> you would put the color scheme you want from<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>/usr/share/vim/vim72/colors</b></span><br />
<br />
It would have colors scheme files like this:<br />
blue.vim, delek.vim, desert.vim, evening.vim etc. etc.<br />
<br />
So when you add it to your .vimrc you don't need the .vim part. Just do:<br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b>:color blue</b></span>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-4941922800881243346.post-50629873276559567492011-06-18T16:25:00.000-04:002011-06-18T16:25:25.615-04:00Change SwappinessI installed CentOS on VirtualBox on my work laptop which has very poor specs. I gave it 512 MB RAM and a single core. Within about 20 minutes the VM started swapping and the whole laptop became extremely slow. I had been researching the problem and came across "<b>swappiness</b>".<br />
<br />
I have changed this setting to zero (0) on this VM. I will test and report back.<br />
<br />
<b><a href="http://lokir.wordpress.com/2010/02/23/swapiness/">What is swappiness and how to change it?</a></b>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-54833146686917672512011-05-29T10:14:00.004-04:002011-05-29T10:54:20.072-04:00CentOS 5.6 Boot Time Error MessagesI installed CentOS 5.6 and was getting the following 2 warning messages at boot time:<br />
<br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b style="background-color: black;">Memory for crash kernel (0x0 to 0x0) notwithin permissible range</b></span><br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b style="background-color: black;">WARNING calibrate_APIC_clock: the APIC timer calibration may be wrong.</b></span><br />
<br />
To fix these, edit /etc/grub.conf and change this line as follows:<br />
<br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b style="background-color: black;">kernel /boot/vmlinuz-2.6.18-238.9.1.el5 ro root=LABEL=/ rhgb apiccalibrationdiff=10000 divider=10</b></span><br />
<br />
So you have to add the following:<br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b style="background-color: black;">apiccalibrationdiff=10000</b></span><br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b style="background-color: black;">divider=10</b></span><br />
<br />
And Remove:<br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b style="background-color: black;">quiet</b></span><br />
<br />
That should do it.<br />
<br />
References:<br />
<a href="http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1032607"><b><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">VMWare KB About APIC Warning Message</span></b></a><br />
<a href="http://www.houseoflinux.com/debates/memory-for-crash-kernel-0x0-to-0x0-notwithin-permissible-range"><b><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">Article to Fix the KDump/KExec Warning at Boot Time</span></b></a>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4941922800881243346.post-26569789271152431592011-03-19T23:18:00.004-04:002012-10-21T20:24:54.425-04:00Quote by Gerald GouldThis was on Criminal Minds.<br />
<b><i><br /></i></b>
<b><i>Beyond the East the sunrise, beyond the West the sea, And the East and West the wander-thirst that will not let me be.</i></b>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-51626831939923491712011-03-16T16:31:00.004-04:002011-03-16T18:15:44.115-04:00Roundcubemail with Postfix on FedoraI just set up Roundcubemail on the Fedora+Postfix server. There is an array of not so great tutorials but I found a good one which is complete. Here is the link:<br />
<br />
<b><a href="http://www.city-fan.org/tips/PaulHowarth/Blog/2008-06-25">Easy Setup of Roundcubemail</a></b><br />
<br />
Came across one issue where mail could not be deleted.<br />
Had to edit the following directive in <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>/etc/roundcubemail/main.inc.php</b></span><br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b style="background-color: black;">$rcmail_config['create_default_folders'] = TRUE;</b></span><br />
<span class="Apple-style-span" style="color: yellow; font-family: 'Courier New', Courier, monospace;"><b><br />
</b></span><br />
<span class="Apple-style-span" style="background-color: black; color: white; font-family: inherit;">Restart all related services.</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-32099238968820379582011-03-14T22:06:00.002-04:002011-03-14T22:07:30.832-04:00Fedora + Postfix + Dovecot + TLS/SSL + SASLI was trying to set up Postfix with Dovecot, TLS/SSL on a new server and came across some issues so I documented them here and the install process again.<br />
<div><br />
</div><div><u><b>Tutorials Used</b></u></div><div><ul><li><a href="http://fedorasolved.org/server-solutions/postfix-mail-server"><b>Fedora Mail Server</b></a></li>
<li><a href="http://wiki.centos.org/HowTos/postfix_sasl"><b>Postfix on CentOS</b></a></li>
</ul></div><div><u><b>Install the Following</b></u></div><div><ul><li>Postfix</li>
<li>Dovecot</li>
<li>OpenSSL</li>
<li>MailX</li>
<li>SASLAuthD</li>
</ul></div><div>Configure Postfix according to my previous article on the blog</div><div><div>In /etc/dovecot/conf.d/10-auth.conf add the following:</div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">passdb<span class="Apple-tab-span" style="white-space: pre;"> </span>{</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>driver = pam</span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></div></div><div><br />
</div><div><div><u><b>Forward these Ports to the Mailserver if Behind Gateway</b></u></div><div>smtp: 25, 2525, 3535</div><div>smtp submission: 587</div><div>smtp over ssl: 465</div><div>pop3: 110, 995</div><div>imap: 143, 220, 993, 585</div></div><div><br />
</div><div>Activate and start all the above daemons using <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">chkconfig</span> and <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">service saslauthd on</span> etc.</div><div><br />
</div><div><div><b><u>Issue #1</u></b></div><div>Ran into a problem where Postfix would not relay mail for its own domain.</div><div>Had to add $mydomain to "mydestination" directive.</div><div><br />
</div><div><b><u>Issue #2</u></b></div><div>/var/log/maillog said mail coming from outside is going to root@platinumfactor.com even though the user exists on the system.</div><div>saslauthd service was not running. Also removed these from main.cf</div><div>smtpd_sasl_type = dovecot</div><div>smtpd_sasl_path = private/auth</div><div><br />
</div><div><b><u>Issue #3</u></b></div><div>Outgoing mail was not being delivered.</div><div>I had enabled TLS and provided a certificate to Postfix.</div><div>The client needed to be told to use SSL/TLS.</div><div><br />
</div><div><b><u>Issue #4</u></b></div><div>Set disable_plaintext_auth = yes</div><div>Incoming mail not being delivered.</div><div>Set client to use Secure POP (port 995)</div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-28878896328726234832011-03-13T20:38:00.005-04:002011-03-22T20:12:14.416-04:00IPTables Port ForwardingSince the creation of my new virtual network, I use one VM just to NAT/route packets so everything else is behind that gateway. I was recently searching on how to do port forwarding to the Postfix behind the gateway and I came across several articles that essentially say the same thing. There are just 2 steps to it assuming you have IPTables up and running:<br />
<br />
<br />
<span class="Apple-style-span" style="background-color: black; color: yellow; font-size: x-small;"><b>iptables -A PREROUTING -t nat -i eth1 -p tcp --dport 80 -j DNAT --to 192.168.1.50:80</b></span><br />
<span class="Apple-style-span" style="background-color: black; color: yellow; font-size: x-small;"><b>iptables -A INPUT -p tcp -m state --state NEW --dport 80 -i eth1 -j ACCEPT</b></span><br />
<br />
then to save the rules:<br />
<span class="Apple-style-span" style="background-color: black; color: yellow; font-size: x-small;"><b>service iptables save</b></span><br />
<ul><li>eth1 is the external interface that accepts packets</li>
<li>192.168.1.50 is the IP address where packets are forwarded</li>
<li>Port 80 is the port that is being forwarded</li>
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-25220968697536432262011-03-02T18:45:00.004-05:002014-09-29T22:48:37.571-04:00IPTables NAT / IP Masquerading<br />
I have been searching for a decent tutorial as I don't know much IPTables. I just needed to set up a simple NAT on my VM Network, so I can run a DHCP server internally to test Cobbler.<br />
<br />
This tutorial is bang on and does the job:<br />
<br />
<a href="http://www.howtoforge.com/nat_iptables"><b>Configuring NAT with IPTables</b></a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-61424405694350476352011-01-18T16:36:00.001-05:002012-03-24T22:09:23.994-04:00Quote by Eliphas Levi<b><i>To attain Sanctum Regnum, there are four indispensable conditions - an intelligence illuminated by study, an intrepidity which nothing can check, a will which nothing can break, and a discretion which nothing can corrupt and nothing intoxicate.</i></b>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4941922800881243346.post-80980604433374452632011-01-14T22:08:00.002-05:002011-01-14T22:09:07.249-05:00Installing and Configuring Squid on Fedora 14Start with, as usual: <span class="Apple-style-span" style="background-color: black; color: yellow; font-family: 'Courier New', Courier, monospace;"><b>yum install squid</b></span><br />
<br />
The main configuration file is in /etc/squid/squid.conf<br />
<br />
The default configuration should be okay for most scenarios.<br />
However, here is how to configure:<br />
<b><a href="http://www.deckle.co.za/squid-users-guide/Squid_Configuration_Basics">Squid Configuration Basics</a></b><br />
<br />
Use these links to configure reverse proxy on Squid:<br />
<a href="http://wiki.squid-cache.org/SquidFaq/ReverseProxy"><b>Official Squid Reverse Proxy Guide</b></a><br />
<a href="http://www.visolve.com/squid/whitepapers/reverseproxy.php"><b>Visolve Squid Reverse Proxy Guide</b></a>Unknownnoreply@blogger.com2