This blog is archived. Please check out the latest news here.
https://catswords.social/@gnh1201
Thank you 🙂
This blog is archived. Please check out the latest news here.
https://catswords.social/@gnh1201
Thank you 🙂

How to use Elasticsearch and ELK (Elasticsearch + Logstash + Kibana) more strongly!
This tool is a transport layer (L4) based data migration tool for MySQL to Elastic.
Successfully transfer data from the relational database (RDBMS) to the Elastic Stack by replacing the L7 (application layer) -based data migration tool with the transport layer (L4).
Data can be migrated 200 to 10,000 times faster than conventional methods.
Whether you are planning to develop your own migration tool, or have developed a migration tool, but are still experiencing unsatisfactory performance.
* I need these people
1. If you used a message queue (Redis, Kafka, RabbitMQ, etc.) to transfer data from MySQL (or MariaDB) to the Elastic Stack
=> No programming effort required for message queues when using L4, avoiding the expectation and other adverse effects (slowing down) due to message queues
In order to achieve the purpose of 2.1, I confirmed the SDK document of MySQL and Elastic Stack directly and implements it with Python, Node.js and so on.
=> No need to know Elastic SDK when using L4
To accomplish the purpose of 3.1, convert MySQL data to JSON file, save it, and use Logastash Tail function or Elastic Filebeat.
=> No need to know complex functions of ELK when using L4
If you have used JDBC to accomplish the purpose of Section 4.1, but you are burdened with too high a memory footprint, long-term connection retention, and frequent migration failures (ex: logstash-input-jdbc)
=> Can use less than 1GB of memory when using L4
5. If you have been looking for a separate migration tool to achieve the goal of 1 (ex: go-mysql-elasticsearch)
=> There is no need to look for a separate migration tool when using L4
6. I have made the output of 2, 3, 4, 5, 6 to achieve the purpose of 1, but the data migration rate is very slow. (About 1,000 ~ 10,000 / day when using L7)
=> 200 ~ 10,000 times amplification when using L4
The L4 Data Migration Tool (MySQL to Elastic) reflects the migration issues that occurred during the MySQL to Elastic process from four companies.
* Characteristic
1. Ultra-fast RDBMS to Elastic (type: key-value / JSON) data migration based on L4
2. LAST ID tracking function (Prevent data duplication migration)
3. Support RPC (Remote Procedure Call) based on REST API
4. Immediate comparison of performance with and without L4-based migration. Compare with the stack of Elastic official documents!
Use Catswords Research’s L4 Data Migration Tool (MySQL to Elastic) for a successful data migration!
* Oracle DB Option
Oracle DB is also available. Please select from more options. Oracle2ES over L4 can be used like Oracle Golden Gate.
price: $450
contact: gnh1201@gmail.com
PHPMailer 5.1, STMP, QSMTPD
[Options] PHPMailer 5.1: $mail->SMTPDebug = 4, PHP 7.0: display_errors->on, error_reporting->E_ALL
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “220 mail.example.org ESMTP
SMTP -> get_lines(): $data is “220 mail.example.org ESMTP
SMTP -> FROM SERVER:220 mail.example.org ESMTP
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “250-mail.example.org Hi Unknown [192.168.0.101]
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
SMTP -> get_lines(): $data was “250-mail.example.org Hi Unknown [192.168.0.101]
SMTP -> get_lines(): $str is “250-PIPELINING
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
SMTP -> get_lines(): $data was “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
SMTP -> get_lines(): $str is “250-8BITMIME
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
SMTP -> get_lines(): $data was “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
SMTP -> get_lines(): $str is “250 STARTTLS
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
250 STARTTLS
SMTP -> FROM SERVER: 250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
250 STARTTLS
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “500 Unrecognized command
SMTP -> get_lines(): $data is “500 Unrecognized command
SMTP -> ERROR: AUTH not accepted from server: 500 Unrecognized command
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “221 mail.example.org closing connection. Have a wonderful day.
SMTP -> get_lines(): $data is “221 mail.example.org closing connection. Have a wonderful day.
SMTP -> FROM SERVER:221 mail.example.org closing connection. Have a wonderful day.
SMTP Connect() failed.
or
500 TLS Negotiation Failed (when tls://mail.example.org)
Warning: stream_socket_enable_crypto(): SSL operation failed with code 1. OpenSSL Error messages:
error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed in /var/www/public/modules/Emails/class.smtp.php (in PHP 7.0, PHPMailer)
You have to test manually e-mail sending with authorization on TLS.
perl -MMIME::Base64 -e ‘print encode_base64(“\admin\@example.org\000fake.password”)’
B2RtaW5AZXhhbXBsZS5vcmcAZmFrZS5wYXNzd29yZA==
$ telnet mail.example.org 587
Trying 127.0.0.1…
Connected to mail.example.org.
Escape character is ‘^]’.
220 mail.example.org ESMTP
ehlo testing
250-mail.example.org Hi Unknown [192.168.0.102]
250-PIPELINING
250-8BITMIME
250 STARTTLS
$ openssl s_client -starttls smtp -crlf -connect mail.example.org:587
CONNECTED(00000003)
depth=1 C = XY, ST = unknown, L = unknown, O = QSMTPD, OU = CA, CN = 220e85afd863, emailAddress = postmaster@220e85afd863
verify error:num=19:self signed certificate in certificate chain
verify return:0
—
Certificate chain
0 s:/C=XY/ST=unknown/L=unknown/O=QSMTPD/OU=Server/CN=220e85afd863/emailAddress=postmaster@220e85afd863
i:/C=XY/ST=unknown/L=unknown/O=QSMTPD/OU=CA/CN=220e85afd863/emailAddress=postmaster@220e85afd863
1 s:/C=XY/ST=unknown/L=unknown/O=QSMTPD/OU=CA/CN=220e85afd863/emailAddress=postmaster@220e85afd863
i:/C=XY/ST=unknown/L=unknown/O=QSMTPD/OU=CA/CN=220e85afd863/emailAddress=postmaster@220e85afd863
—
Server certificate
—–BEGIN CERTIFICATE—–
MIIDcjCCAlqgAwIBAgIBAjANBgkqhkiG9w0BAQsFADCBjjELMAkGA1UEBhMCWFkx
EDAOBgNVBAgMB3Vua25vd24xEDAOBgNVBAcMB3Vua25vd24xDzANBgNVBAoMBlFT
TVRQRDELMAkGA1UECwwCQ0ExFTATBgNVBAMMDDIyMGU4NWFmZDg2MzEmMCQGCSqG
SIb3DQEJARYXcG9zdG1hc3RlckAyMjBlODVhZmQ4NjMwHhcNMTcwOTE5MDUyMDI0
WhcNMTkwOTE5MDUyMDI0WjCBkjELMAkGA1UEBhMCWFkxEDAOBgNVBAgMB3Vua25v
d24xEDAOBgNVBAcMB3Vua25vd24xDzANBgNVBAoMBlFTTVRQRDEPMA0GA1UECwwG
U2VydmVyMRUwEwYDVQQDDAwyMjBlODVhZmQ4NjMxJjAkBgkqhkiG9w0BCQEWF3Bv
c3RtYXN0ZXJAMjIwZTg1YWZkODYzMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB
gQDHKMpOZTEjJdJenoD1bgNJ5+mH+xuC0bQTlDIUSYDP79CBa9q6y1hSrT3e8POQ
+hDQccA4WCFO6sUlfkKCdyCMOMitmnfuMrOV2tmECntX3lUGZasASg+3HVTz2Bdb
bNeJObX9jLZNBOOZF/14vff5Y7eYJKq15bNv3ISWqLEkawIDAQABo1kwVzAiBgNV
HREEGzAZgRdwb3N0bWFzdGVyQDIyMGU4NWFmZDg2MzAeBglghkgBhvhCAQ0EERYP
dGxzIGNlcnRpZmljYXRlMBEGCWCGSAGG+EIBAQQEAwIGQDANBgkqhkiG9w0BAQsF
AAOCAQEAK/ctHc6ciSmOxGRrBQ0pu+aMUu8Msl4Noa5U0pid+x8ErktM7ULdncIr
K4zmTeNKSKjciEc7VPtwaATLcWLQQp0y+Xzc+0KxapKsXVsCYaNlfXBzQfTx2vkX
cUtpQ2UqYKFIvOBMv7R+18+rmWj50BV9FxHZli1Y8+usms5tdsH/o18WytP2pyip
lVqla+wmb8synLh3lUuOlvprQSWWp1+Zh0wXVszHmfQmSNQuVlmRleU04IQo5Ye0
kjbiedcwgCT7mt3mi5lihsiNxfx/cvZ/8k512WEg8xA+FEORHoQdszWKUqNAtjZY
D4HyZo0r2S58LuMvQX2hfbPNZi7/kg==
—–END CERTIFICATE—–
subject=/C=XY/ST=unknown/L=unknown/O=QSMTPD/OU=Server/CN=220e85afd863/emailAddress=postmaster@220e85afd863
issuer=/C=XY/ST=unknown/L=unknown/O=QSMTPD/OU=CA/CN=220e85afd863/emailAddress=postmaster@220e85afd863
—
No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits
—
SSL handshake has read 2484 bytes and written 408 bytes
—
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 28C35D87DA8DC1661418259F5CD17456116E67C7CC41C2212FF23508F61220EC
Session-ID-ctx:
Master-Key: 2330D125C16D0131BEA156E62EC3EFF64AF46F2472F839B9F8D6E4BC98F1E56CD1EBC20E95F7E3C13201F126F7D6ED21
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 – 20 e1 a3 6d 87 d5 da 79-d2 30 2e 1c dc 93 bc 93 ..m…y.0……
0010 – 18 8d 8a b7 01 73 b0 71-3a d4 67 60 ef 2c 0b a6 …..s.q:.g`.,..
0020 – 21 3f a3 58 85 43 93 0a-04 52 cd b0 06 d6 af 3b !?.X.C…R…..;
0030 – 9f ff 3b 0f b8 ad 31 8b-50 3f ae 33 21 c0 92 a6 ..;…1.P?.3!…
0040 – b5 2f 52 79 eb b7 61 12-fd be 41 35 4d 14 51 86 ./Ry..a…A5M.Q.
0050 – 96 c6 9a 28 ac 5b 37 ce-db 08 65 0f 57 19 c3 67 …(.[7…e.W..g
0060 – 91 00 f1 33 fc ae 22 c1-89 23 42 a1 07 2e af 7d …3..”..#B….}
0070 – f7 c6 5f 0f d5 b9 47 7f-aa 85 eb 19 79 b5 16 7b .._…G…..y..{
0080 – 99 94 6a 03 15 a7 7f e9-8b a8 6e 8b 8f 74 28 f6 ..j…….n..t(.
0090 – ca 16 01 70 44 d1 72 d7-5b 0d a2 d6 d1 78 26 c2 …pD.r.[….x&.Start Time: 1506060093
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
—
250 STARTTLS
ehlo testing
250-mail.example.org Hi Unknown [192.168.0.102]
250-PIPELINING
250-8BITMIME
250 AUTH LOGIN PLAIN
AUTH PLAIN B2RtaW5AZXhhbXBsZS5vcmcAZmFrZS5wYXNzd29yZA==
235 PLAIN authentication successful for admin@example.org – auth success for admin@example.org
mail from: <admin@example.org>
250 <admin@example.org>, sender OK – how exciting to get mail from you!
rcpt to: <gnh1201@gmail.com>
250 <gnh1201@gmail.com>, recipient ok
data
354 go ahead
From: Admin <admin@example.org>
To: Customer <gnh1201@gmail.com>
Subject: Thank you for your interesting.Hi Customer. Thank you for your interesting.
.
250 Queued! 1506060197 qp 697 <1619abc426b8214dd55ff1829d235633@missing.id>
quit
221 mail.example.org closing connection. Have a wonderful day.
closed
before:
// connect to the smtp server
$this->smtp_conn = @fsockopen($host, // the host of the server
$port, // the port to use
$errno, // error number if any
$errstr, // error message if any
$tval); // give up after ? secs
after:
$socket_context = stream_context_create(array(‘ssl’ => array(‘verify_peer’ => false,’verify_peer_name’ => false,’allow_self_signed’ => true)));
$this->smtp_conn = stream_socket_client($host . ‘:’ . $port,$errno,$errstr,$tval,STREAM_CLIENT_CONNECT,$socket_context);
before:
stream_socket_enable_crypto($this->smtp_conn, true, STREAM_CRYPTO_METHOD_TLS_CLIENT)
after:
//Allow the best TLS version(s) we can
$crypto_method = STREAM_CRYPTO_METHOD_TLS_CLIENT;
//PHP 5.6.7 dropped inclusion of TLS 1.1 and 1.2 in STREAM_CRYPTO_METHOD_TLS_CLIENT
//so add them back in manually if we can
if (defined(‘STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT’)) {
$crypto_method |= STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT;
$crypto_method |= STREAM_CRYPTO_METHOD_TLSv1_1_CLIENT;
}stream_socket_enable_crypto($this->smtp_conn, true, $crypto_method)
[Options] host->tls://mail.example.org:587, smtp_auth->true
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “220 mail.example.org ESMTP
SMTP -> get_lines(): $data is “220 mail.example.org ESMTP
SMTP -> FROM SERVER:220 mail.example.org ESMTP
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “250-mail.example.org Hi Unknown [192.168.0.101]
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
SMTP -> get_lines(): $data was “250-mail.example.org Hi Unknown [192.168.0.101]
SMTP -> get_lines(): $str is “250-PIPELINING
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
SMTP -> get_lines(): $data was “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
SMTP -> get_lines(): $str is “250-8BITMIME
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
SMTP -> get_lines(): $data was “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
SMTP -> get_lines(): $str is “250 STARTTLS
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
250 STARTTLS
SMTP -> FROM SERVER: 250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
250 STARTTLS
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “220 Go ahead with TLS
SMTP -> get_lines(): $data is “220 Go ahead with TLS
SMTP -> FROM SERVER:220 Go ahead with TLS
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “250-mail.example.org Hi Unknown [192.168.0.101]
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
SMTP -> get_lines(): $data was “250-mail.example.org Hi Unknown [192.168.0.101]
SMTP -> get_lines(): $str is “250-PIPELINING
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
SMTP -> get_lines(): $data was “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
SMTP -> get_lines(): $str is “250-8BITMIME
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
SMTP -> get_lines(): $data was “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
SMTP -> get_lines(): $str is “250 AUTH PLAIN LOGIN
SMTP -> get_lines(): $data is “250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
250 AUTH PLAIN LOGIN
SMTP -> FROM SERVER: 250-mail.example.org Hi Unknown [192.168.0.101]
250-PIPELINING
250-8BITMIME
250 AUTH PLAIN LOGIN
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “334 VXNlcm5hbWU6
SMTP -> get_lines(): $data is “334 VXNlcm5hbWU6
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “334 UGFzc3dvcmQ6
SMTP -> get_lines(): $data is “334 UGFzc3dvcmQ6
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “235 LOGIN authentication successful for admin@example.org – auth success for admin@example.org
SMTP -> get_lines(): $data is “235 LOGIN authentication successful for admin@example.org – auth success for admin@example.org
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “250 <admin@example.org>, sender OK – how exciting to get mail from you!
SMTP -> get_lines(): $data is “250 <admin@example.org>, sender OK – how exciting to get mail from you!
SMTP -> FROM SERVER:250 <admin@example.org>, sender OK – how exciting to get mail from you!
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “250 <customer@example.org>, recipient ok
SMTP -> get_lines(): $data is “250 <customer@example.org>, recipient ok
SMTP -> FROM SERVER:250 <customer@example.org>, recipient ok
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “354 go ahead
SMTP -> get_lines(): $data is “354 go ahead
SMTP -> FROM SERVER:354 go ahead
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “250 Queued! 1506308867 qp 14594 <28ef7c771b6ac10e8bd5b252c1e56386@demo.example.org>
SMTP -> get_lines(): $data is “250 Queued! 1506308867 qp 14594 <28ef7c771b6ac10e8bd5b252c1e56386@demo.example.org>
SMTP -> FROM SERVER:250 Queued! 1506308867 qp 14594 <28ef7c771b6ac10e8bd5b252c1e56386@demo.example.org>
SMTP -> get_lines(): $data was “”
SMTP -> get_lines(): $str is “221 mail.example.org closing connection. Have a wonderful day.
SMTP -> get_lines(): $data is “221 mail.example.org closing connection. Have a wonderful day.
SMTP -> FROM SERVER:221 mail.example.org closing connection. Have a wonderful day.
Well done! Congratulations!
Problems:
Solution:
user1:x:1000:1000::/home/user1:/sbin/nologin
user2:x:1001:1001::/home/user2:/bin/bash
You have to change /sbin/nologin to /bin/bash. and save it!
See also:
username@hostname:~$ sudo apt-get install php7.0-xml
username@hostname:~$ sudo service apache2 restart
username@hostname:~$ php -m
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib/php/20151012/dom.so’ – /usr/lib/php/20151012/dom.so: undefined symbol: php_libxml_node_free_list in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib/php/20151012/xmlreader.so’ – /usr/lib/php/20151012/xmlreader.so: undefined symbol: dom_node_class_entry in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib/php/20151012/xsl.so’ – /usr/lib/php/20151012/xsl.so: undefined symbol: dom_node_class_entry in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib/php/20151012/dom.so’ – /usr/lib/php/20151012/dom.so: undefined symbol: php_libxml_node_free_list in Unknown on line 0
(In Ubuntu 14.04 or above)
Step 1:
sudo apt-get install libapache2-mod-php7.0 libphp7.0-embed libssl-dev openssl php7.0-cgi php7.0-cli php7.0-common php7.0-dev php7.0-fpm php7.0-phpdbg
Step 2:
sudo service apache2 restart
Quick command (copy & paste):
sudo apt-get install libapache2-mod-php7.0 libphp7.0-embed libssl-dev openssl php7.0-cgi php7.0-cli php7.0-common php7.0-dev php7.0-fpm php7.0-phpdbg php7.0-xml && sudo service apache2 restart
https://bugs.php.net/bug.php?id=74865
# chsh [-s login_shell] [user] (example: chsh -s /bin/bash username)
chsh: PAM authentication failed
This error can be occur by invalid shadow password configuration. (/etc/passwd)
Generally, chsh command is run with root privilege.
but, It have riskiness that change login shell of root user if you wasn’t input username with invalid login shell path.
(Fatal example: # chsh -s /home/username [empty])
You have to check /etc/password configuration file.
# vi /etc/password
root:x:0:0:root:/root:/home/username
this part (/home/username) modify to /bin/bash (or your login shell)
Next, you try check chsh command.
Related post: VirtualBox: What is difference OSE vs FUEL? and following troubles.
If you want remove VirtualBox in order to solve something issues, you have to remove virtualbox completely.
but, sometimes(or always), package managers (such as APT, YUM, etc) can not remove VirtualBox completely. so, I wrote uninstall guide in this post.
This post based on Debian/Ubuntu distribution.
Thanks.
If you want find snort db schema, maybe you can not.
You have to check barnyard2 project: https://github.com/firnsy/barnyard2
Base on Debian Jessie or Ubuntu 16.04:
How to troubleshooting:
You can install snort! Enjoy.
References: