分类： Script

使用rsync定期同步windows系统文件到Linux系统

1 Windows 平台 cwRsync 安装配置成客户端，通过设置计划任务每 5 分钟推文件到 linux 的 rsync 服务端。
2 linux 配置成服务端。
3 记录同步详细日志，linux平台记录日志。
4 只同步新增或者修改过的不一样的文件，忽略时间、权限的同步。
示意图如下：

+----------------------+            |           +----------------------+
|   Windows IIS web    |10.10.100.2 |10.10.100.0|   Linux Nginx web    |
|                      +------------+-----------+                      |
|   /data/htdocs/*     |    --------------->    |     /data/htdocs/*   |
+----------------------+         copy           +----------------------+

一、Windows 客户端配置：
cwRsync是基于cygwin平台的rsync软件包，支持windows对windows、windows对Linux、Linux对windows高效文件同步。由于CwRsync已经集成了cygwin类库，因此安装的时候可以省去cygwin包。Cwrsync还集成了OpenSSH for windows，可以实现Linux 下Rsync一模一样的操作。使用 cwRsync 来同步文件后，只需要对一台主服务器进行文件修改，其他镜像服务器可以自动同步，包括文件的更新、删除、重命名等。

cwRsync分为付费版和免费版两种，我们只需要使用免费版即可，在官方网站上面下载 cwRsync Free Edition 版本。

https://www.itefix.net/content/cwrsync-free-edition
Name: cwRsync_5.5.0_x86_Free.zip
SHA256: 37e8ef21ac975d4ee86c9d3be40c8935e8b9d0ba84e9302fc106b9452296cb85

包含如下几个程序
Version information:
Rsync 3.1.2
Cygwin 2.3.1
OpenSSH 7.1p
OpenSSL 1.0.2e

1.2 解压 cwRsync_5.5.0_x86_Free.zip 到 D:\data\app\cwRsync_5.5.0_x86_Free 目录中
1.3 双击 cwrsync.cmd 运行，会在当前生成 home\%USERNAME%\.ssh 目录，供 ssh 认证方式使用。
1.4 为系统新建一个环境变量，目录为cwrsync的bin目录下，例：path:D:\data\app\cwRsync_5.5.0_x86_Free\bin。这样 cmd 命令行下可以直接运行 rsync 这个命令
C:\Users\Administrator>rsync –version
rsync version 3.1.2 protocol version 31

2.1 Ubunut 16.04 下安装 rsync

apt-get install rsync
# rsync  --version
rsync  version 3.1.1  protocol version 31

2.2 配置文件
vim /etc/rsyncd.conf

pid file = /var/run/rsync.pid
port = 873
log file = /var/log/rsyncd.log
lock file = /var/run/rsync.lock 

# create new
# any name you like
[resources]
# destination directory to copy
path = /data/htdocs/
# hosts you allow to access
hosts allow = 10.10.100.2
hosts deny = *
secrets file = /etc/rsyncd.pass
# 表示该节点是否可被发现。
list = true
# 指定传输到这里文件所属的用户。
uid = www-data
# 指定传输到这里的文件所属的组。
gid = www-data
# rsync连接时的用户名，要和客户端rsync的命令一致
auth users = www-data
# 该目录是否只读
read only = no
transfer logging = yes
log format = %t: host %h (%a) %o %f (%l bytes). Total %b bytes.  
timeout = 600

# 创建密码文件
touch /etc/rsyncd.pass
#权限修改
chown root:root /etc/rsyncd.pass
chmod 600 /etc/rsyncd.pass
vim /etc/rsyncd.pass
内容为：www-data:passwd # 用户名：密码

2.3 启动服务端

service rsync start

# 添加开机启动
update-rc.d rsync defaults

2.4 检查 rsync 端口，确认已启动

netstat -ntlpu | grep rsync
tcp        0      0 0.0.0.0:873             0.0.0.0:*               LISTEN      25753/rsync     
tcp6       0      0 :::873                  :::*                    LISTEN      25753/rsync

3.1 客户端测试
新建立目录
D:\data\app\cwRsync_5.5.0_x86_Free\etc
创建 rsync.pass 文件
输入 rsync 服务端的密码

# -u 同步不覆盖(源到目的)目的产生的新文件内容,不修改覆盖目标目录下的新增类的文件
完整的同步命令：

rsync -acrtzv --password-file=/cygdrive/d/data/app/cwRsync_5.5.0_x86_Free/etc/rsync.pass /cygdrive/d/data/htdocs/ www-data@10.10.100.0::resources

日志：
2017/02/27 14:44:38 [25798] name lookup failed for 10.10.100.2: Name or service not known
解决这个 not known 的问题，把客户端的 hostname 写入服务端的系统 host 文件中

vim /etc/hosts

10.10.100.2 IISWEB01

3.2 创建同步脚本文件
D:\data\app\cwRsync_5.5.0_x86_Free\etc
dynamicres_sync.bat

@echo off
D:\data\app\cwRsync_5.5.0_x86_Free\bin\rsync -acrtzv --password-file=/cygdrive/d/data/app/cwRsync_5.5.0_x86_Free/etc/rsync.pass /cygdrive/d/data/htdocs/ www-data@10.10.100.0::resources

4.1 在Windows中创建任务计划：
每5分钟执行一次 dynamicres_sync.bat 这个同步脚本

5.1 Linux Rsync server 日志
发现日志中第二行和第三行时间是使用 UTC 时间。

# tail -100f /var/log/rsyncd.log 
2017/02/27 16:15:58 [26150] connect from IISWEB01 (10.10.100.2)
2017/02/27 08:15:58 [26150] rsync to resources/ from www-data@IISWEB01 (10.10.100.2)
2017/02/27 08:15:58 [26150] receiving file list
2017/02/27 08:15:59 [26150] sent 25 bytes  received 7441 bytes  total size 8515857
2017/02/27 16:20:58 [26156] connect from IISWEB01 (10.10.100.2)
2017/02/27 08:20:58 [26156] rsync to resources/ from www-data@IISWEB01 (10.10.100.2)
2017/02/27 08:20:58 [26156] receiving file list
2017/02/27 08:20:59 [26156] sent 25 bytes  received 7441 bytes  total size 8515857

update 20171028 15:05
解决 rsync 同步到 Linux 系统时，目录及文件权限不正常

-artzv

-a, ––archive 归档模式，表示以递归方式传输文件，并保持所有文件属性，等价于 -rlptgoD (注意不包括 -H)

-r, ––recursive 对子目录以递归模式处理
-l 保持符号链接文件
#-p 保持文件权限
-t 保持文件时间信息
#-g 保持文件归属组信息
#-o 保持文件归属用户信息
-D 保持设备文件和特殊文件

rltDzv 需要使用这条命令，把 -p -g -o权限都去掉

目录权限：755
文件：644
用户组： 为 rsyncd.conf 中指定的。
# 指定传输到这里文件所属的用户。
uid = www-data
# 指定传输到这里的文件所属的组。
gid = www-data

修改后的命令为：

D:\data\app\cwRsync_5.5.0_x86_Free\bin\rsync -rltDzv --chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r --password-file=/cygdrive/d/data/app/cwRsync_5.5.0_x86_Free/etc/rsync.pass /cygdrive/d/data/htdocs/ www-data@10.10.100.0::resources

update 2018/10/16
windows rsync 通过 ssh 端口及认证传送文件到 linux 服务器下
此方式远程服务器不需要运行rsync-demo，但必须安装 rsync 客户端
配置ssh key
创建 .ssh 目录，做好ssh-key免密码登录

C:\cwRsync\home\Administrator\.ssh

C:\cwRsync\bin\rsync.exe -rtv -e "/cygdrive/c/cwRsync/bin/ssh.exe -p 22" /cygdrive/d/data/ftp/* kye00@10.10.100.0:/data/ftp/
或者指定key
C:\cwRsync\bin\rsync.exe -rtv -e "/cygdrive/c/cwRsync/bin/ssh.exe -p 22 -i /cygdrive/c/xxx/id_rsa " /cygdrive/d/data/ftp/* kye00@10.10.100.0:/data/ftp/

windows 版免费 server 端
github 地址
https://github.com/backuppc/cygwin-rsyncd

一，先安装 脚本工具
nsis-3.01-setup.exe

2
右键 backuppc_rsync-server.nsi
会生成
cygwin-rsyncd-3.1.2.1_installer.exe 安装工具

3 运行 cygwin-rsyncd-3.1.2.1_installer.exe 安装工具
3.1 会自动复制各个程序到 C:\rsyncd 目录下
3.1 会自动把Rsyncd添加到服务里面启动。

4 修改配置
c:\rsyncd\rsyncd.conf and c:\rsyncd\rsyncd.secrets files to set your client-specific shares,
backup user name and password.
重启rsyncd 服务才会生效
restart the Windows RsyncServer service to get the new settings.

5 配置文件权限 rsyncd.secrets 去掉 user用户组，只保留administrator 和 system
rsyncd.conf 修改限制IP项目

To ensure initial security, the c:\rsyncd\rsyncd.secrets file initially has no users, and the c:\rsyncd\rsyncd.conf only allows connections from two specific IP addresses. So unless you edit those two files you won’t be able to connect to the rsyncd server.

6、开放 873 内网访问
If you have Windows firewall enabled then you will need to allow rsync to listen on TCP port 873. You can do that through the WinXX firewall menus. You can also make that rule specific to the BackupPC server IP addresses, so no other hosts can contact the rsyncd server on this client.

记录 Linux Iptables 防火墙 Dropped Packets 日志

最近VPS深受垃圾评论所害，而且IP都是福建莆田的，而且是不停的访问一两个URL！

以下是屏蔽一个IP列表的脚本中的一段,完整的脚本Google一下就出来了。

IPT="/sbin/iptables"
SPAMLIST="blockedip"
SPAMDROPMSG="BLOCKED IP DROP: "

[ -f /root/scripts/blocked.ips.txt ] && BADIPS=$(egrep -v -E "^#|^$" /root/scripts/blocked.ips.txt)

PUB_IF="eth0"

if [ -f /root/scripts/blocked.ips.txt ];
then
# create a new iptables list
$IPT -N $SPAMLIST

for ipblock in $BADIPS
do
$IPT -A $SPAMLIST -s $ipblock -j LOG --log-prefix "$SPAMDROPMSG"
$IPT -A $SPAMLIST -s $ipblock -j DROP
done

$IPT -I INPUT -j $SPAMLIST
$IPT -I OUTPUT -j $SPAMLIST
$IPT -I FORWARD -j $SPAMLIST
fi

This article is part of our ongoing Linux IPTables series of articles. When things are not working as expected with your IPTables rules, you might want to log the IPTables dropped packets for troubleshooting purpose. This article explains how to log both incoming and outgoing dropped firewal packets.

If you are new to IPTables, first get yourself comfortable with the IPTables fundamental concepts.

Log All Dropped Input Packets

First we need to understand how to log all the dropped input packets of iptables to syslog.

If you already have whole bunch of iptables firewall rules, add these at the bottom, which will log all the dropped input packets (incoming) to the /var/log/messages

iptables -N LOGGING
iptables -A INPUT -j LOGGING
iptables -A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
iptables -A LOGGING -j DROP

In the above example, it does the following:

  • iptables -N LOGGING: Create a new chain called LOGGING
  • iptables -A INPUT -j LOGGING: All the remaining incoming packets will jump to the LOGGING chain
  • line#3: Log the incoming packets to syslog (/var/log/messages). This line is explained below in detail.
  • iptables -A LOGGING -j DROP: Finally, drop all the packets that came to the LOGGING chain. i.e now it really drops the incoming packets.

In the line#3 above, it has the following options for logging the dropped packets:

  • -m limit: This uses the limit matching module. Using this you can limit the logging using –limit option.
  • –limit 2/min: This indicates the maximum average matching rate for logging. In this example, for the similar packets it will limit logging to 2 per minute. You can also specify 2/second, 2/minute, 2/hour, 2/day. This is helpful when you don’t want to clutter your log messages with repeated messages of the same dropped packets.
  • -j LOG: This indicates that the target for this packet is LOG. i.e write to the log file.
  • –log-prefix “IPTables-Dropped: ” You can specify any log prefix, which will be appended to the log messages that will be written to the /var/log/messages file
  • –log-level 4 This is the standard syslog levels. 4 is warning. You can use number from the range 0 through 7. 0 is emergency and 7 is debug.

Log All Dropped Outgoing Packets

This is same as above, but the 2nd line below has OUTPUT instead of INPUT.

iptables -N LOGGING
iptables -A OUTPUT -j LOGGING
iptables -A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
iptables -A LOGGING -j DROP

Log All Dropped Packets (both Incoming and Outgoing)

This is same as before, but we’ll be taking the line number 2 from the previous two examples, and adding it here. i.e We’ll have a separate line for INPUT and OUTPUT which will jump to LOGGING chain.

To log both the incoming and outgoing dropped packets, add the following lines at the bottom of your existing iptables firewall rules.

iptables -N LOGGING
iptables -A INPUT -j LOGGING
iptables -A OUTPUT -j LOGGING
iptables -A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
iptables -A LOGGING -j DROP

Also, as we explained earlier, by default, the iptables will use /var/log/messages to log all the message. If you want to change this to your own custom log file add the following line to /etc/syslog.conf

kern.warning /var/log/custom.log

How to read the IPTables Log

The following is a sample of the lines that was logged in the /var/log/messages when an incoming and outgoing packets was dropped.

Aug 4 13:22:40 centos kernel: IPTables-Dropped: IN= OUT=em1 SRC=192.168.1.23 DST=192.168.1.20 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=59228 SEQ=2
Aug 4 13:23:00 centos kernel: IPTables-Dropped: IN=em1 OUT= MAC=a2:be:d2:ab:11:af:e2:f2:00:00 SRC=192.168.2.115 DST=192.168.1.23 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=9434 DF PROTO=TCP SPT=58428 DPT=443 WINDOW=8192 RES=0x00 SYN URGP=0

In the above output:

  • IPTables-Dropped: This is the prefix that we used in our logging by specifying –log-prefix option
  • IN=em1 This indicates the interface that was used for this incoming packets. This will be empty for outgoing packets
  • OUT=em1 This indicates the interface that was used for outgoing packets. This will be empty for incoming packets.
  • SRC= The source ip-address from where the packet originated
  • DST= The destination ip-address where the packets was sent to
  • LEN= Length of the packet
  • PROTO= Indicates the protocol (as you see above, the 1st line is for outgoing ICMP protocol, the 2nd line is for incoming TCP protocol)
  • SPT= Indicates the source port
  • DPT= Indicates the destination port. In the 2nd line above, the destination port is 443. This indicates that the incoming HTTPS packets was dropped

From http://www.thegeekstuff.com/2012/08/iptables-log-packets/

use expect SSH Auto login

一、要安装这两个东东。
CentOS下

1
yum install tcl
tcl.x86_64 1:8.5.7-6.el6
2
yum install expect
expect.x86_64 0:5.44.1.15-2.el6

ArchLinux下

1
pacman -S tcl
2
pacman -S expect

自动登了ssh脚本

vim sshlogin.exp

 

#!/usr/bin/expect -f
#auto ssh login

set timeout 20
spawn ssh -qTfnN -D 7070 你的登录用户@服务器ip地址

expect "*password:"
send "you passwd\r"
interact

chmod +x sshlogin.exp
二、实现开机自动登录到tty4。
以下是在ArchLinux上操作的，其它系统也差不多。
1、安装自动登录tty的软件

pacman -S mingetty

mingetty-1.08-3-x86_64

vi /etc/inittab

找到关于tty4的，在Archlinux里是c4:2345:respawn:/sbin/agetty…这样的一行改成

c4:23:respawn:/sbin/mingetty --autologin javasboy tty4

把javasboy换成你要登录的用户，且能运行上面脚本的权限。
2、在javasboy用户下的~/.bashrc里面加上：

tty | grep -q tty4 && /home/javasboy/bin/sshlogin.exp

这样tty4就能开机运行这个命令了。

还可以通过crond定时检测7070端口和ssh进程是否运行来重新启动，这样就能实现断线自动重连的功能了。
在此感谢SUN的无私帮助。

每天自动备份mysql脚本

定时执行脚本：
1、
执行

crontab -e
00 00 * * * /bin/bash yourpath/mysqlbak.sh

2、
打开自动执行文件

vi /etc/crontab

在etc中加入如下内容，让其自动执行任务。

00 00 * * * root /mysqlbak.sh

以上两个 00    00    *    *    *  为每天的凌晨自动执行脚本

分 时 日 月 周 命令

M: 分钟（0-59）。每分钟用*或者 */1表示
H：小时（0-23）。（0表示0点）
D：天（1-31）。
m: 月（1-12）。
d: 一星期内的天（0~6，0为星期天）。

每五分钟执行    */5 * * * *
每小时执行      0 * * * *
每天执行        0 0 * * *
每周执行        0 0 * * 0
每月执行        0 0 1 * *
每年执行        0 0 1 1 *

重启cron

/etc/rc.d/init.d/crond restart

or

service crond restart

详细请看crond的wiki
http://zh.wikipedia.org/wiki/Cron

mysqlback.sh

#!/bin/bash
#功能说明：本功能用于备份数据库
#编写日期：2010/12/06

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/local/mysql/bin
export PATH
#数据库用户名
dbuser='root'
#数据库密码
dbpasswd='123456'
#数据库名,可以定义多个数据库，中间以空格隔开，如：test test1 test2
dbname='test1 test2'
#备份时间
backtime=`date +%Y%m%d%H%M%S`
#日志备份路径
logpath='/second/backup'
#数据备份路径
datapath='/second/backup'
#日志记录头部
echo ‘"备份时间为${backtime},备份数据库表 ${dbname} 开始" >> ${logpath}/mysqllog.log
#正式备份数据库
for table in $dbname; do
source=`mysqldump -u ${dbuser} -p${dbpasswd} ${table}> ${logpath}/${backtime}.sql` 2>> ${logpath}/mysqllog.log;
#备份成功以下操作
if [ "$?" == 0 ];then
cd $datapath
#为节约硬盘空间，将数据库压缩
tar jcf ${table}${backtime}.tar.bz2 ${backtime}.sql > /dev/null
#删除原始文件，只留压缩后文件
rm -f ${datapath}/${backtime}.sql
echo "数据库表 ${dbname} 备份成功!!" >> ${logpath}/mysqllog.log
else
#备份失败则进行以下操作
echo "数据库表 ${dbname} 备份失败!!" >> ${logpath}/mysqllog.log
fi
done

这里有一篇介绍
MySQL数据库备份的10个教程

http://www.linuxde.net/2012/03/9379.html