Skip to content
This repository was archived by the owner on Jan 8, 2019. It is now read-only.

Allow MySQL to be served by HAProxy again #1275

Merged
merged 3 commits into from
Sep 26, 2018

Conversation

cbaenziger
Copy link
Member

This should fix the issues mentioned in #1274 . Running test over night to ensure a VM cluster builds.

@leochen4891
Copy link
Contributor

leochen4891 commented Sep 24, 2018

It is definitely not correct to have mysql listening on management ip my.cnf while haproxy is configured using floating ip.
Is there any guidelines for which services should use management ip?
Is it like any no-user-data traffic (configuration, telemetry, admin ... etc ) should go through management ip, or only high priority admin traffic is allowed?

@cbaenziger
Copy link
Member Author

One of the easiest ways to see what the structure was before #1269 is to run netstat -nlp; on a cluster I have handy I see:

ubuntu@rebase-bcpc-vm1:~$ sudo netstat -nl4p | awk '/192\.|10\./ {print $4 "|" $7 }'|sort 
10.0.105.11:10050|11720/zabbix_agentd
10.0.105.11:123|
10.0.105.11:1936|4945/haproxy
10.0.105.11:22|15352/sshd
10.0.105.11:3306|9708/mysqld
10.0.105.11:443|4629/fcgi-pm
10.0.105.11:4567|9708/mysqld
10.0.105.11:80|4629/fcgi-pm
10.0.105.11:8888|4629/fcgi-pm
10.0.105.5:11211|4945/haproxy
10.0.105.5:1936|4945/haproxy
10.0.105.5:2013|18276/python
10.0.105.5:2014|18276/python
10.0.105.5:3306|4945/haproxy
10.0.105.5:7777|4629/fcgi-pm
10.0.105.5:8888|4629/fcgi-pm
192.168.105.11:123|
192.168.105.11:2003|17659/python
192.168.105.11:2004|17659/python
192.168.105.11:2023|18186/python
192.168.105.11:2024|18186/python
192.168.105.11:2181|31966/java
192.168.105.11:3888|31966/java
192.168.105.11:50070|21689/java
192.168.105.11:7002|17659/python
192.168.105.11:8019|10934/java
192.168.105.11:8020|21689/java
192.168.105.11:8480|1293/java
192.168.105.11:8485|1293/java
192.168.105.5:11010|4945/haproxy

You can see I have most all cluster services listening on the floating IP or floating VIP; while cluster-internal services or management services which tenants do not need to connect to are listening on the management IP. (Particularly, MySQL is listening on the management VIP.)

The design is supposed to be pretty simple that management services not intended for tenant usage end up on the management interface while the floating interface is our "service" interface. (Renaming it this way in our code and getting rid of the deprecated storage interface would be helpful to make this more obvious I now see.)

@cbaenziger
Copy link
Member Author

I can confirm I was able to build master with this change.


# links to services (an hash of hashes with structure:
# <name>=>{'desc':<desc> and 'url':<url>} to be listed)
bach_web['links'] = []
Copy link
Contributor

@leochen4891 leochen4891 Sep 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the default value.
I think the comment is correct, and it should be {} rather than [] (reference here)

Although service_ports and files are set by hdfs and hbase attribute file, it might be safer to set the default value here as well, adding

bach_web['service_ports'] = {}
bach_web['files'] = {}

@leochen4891
Copy link
Contributor

One of the easiest ways to see what the structure was before #1269 is to run netstat -nlp; on a cluster I have handy I see:

ubuntu@rebase-bcpc-vm1:~$ sudo netstat -nl4p | awk '/192\.|10\./ {print $4 "|" $7 }'|sort 
10.0.105.11:10050|11720/zabbix_agentd
10.0.105.11:123|
10.0.105.11:1936|4945/haproxy
10.0.105.11:22|15352/sshd
10.0.105.11:3306|9708/mysqld
10.0.105.11:443|4629/fcgi-pm
10.0.105.11:4567|9708/mysqld
10.0.105.11:80|4629/fcgi-pm
10.0.105.11:8888|4629/fcgi-pm
10.0.105.5:11211|4945/haproxy
10.0.105.5:1936|4945/haproxy
10.0.105.5:2013|18276/python
10.0.105.5:2014|18276/python
10.0.105.5:3306|4945/haproxy
10.0.105.5:7777|4629/fcgi-pm
10.0.105.5:8888|4629/fcgi-pm
192.168.105.11:123|
192.168.105.11:2003|17659/python
192.168.105.11:2004|17659/python
192.168.105.11:2023|18186/python
192.168.105.11:2024|18186/python
192.168.105.11:2181|31966/java
192.168.105.11:3888|31966/java
192.168.105.11:50070|21689/java
192.168.105.11:7002|17659/python
192.168.105.11:8019|10934/java
192.168.105.11:8020|21689/java
192.168.105.11:8480|1293/java
192.168.105.11:8485|1293/java
192.168.105.5:11010|4945/haproxy

You can see I have most all cluster services listening on the floating IP or floating VIP; while cluster-internal services or management services which tenants do not need to connect to are listening on the management IP. (Particularly, MySQL is listening on the management VIP.)

The design is supposed to be pretty simple that management services not intended for tenant usage end up on the management interface while the floating interface is our "service" interface. (Renaming it this way in our code and getting rid of the deprecated storage interface would be helpful to make this more obvious I now see.)

Thanks for the explanation.

@cbaenziger
Copy link
Member Author

@leochen4891 Please see if my descriptions match what you were thinking?

@leochen4891
Copy link
Contributor

Thanks for the update. LGTM

@leochen4891 leochen4891 merged commit ad58e08 into bloomberg:master Sep 26, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants