Tachyon如何进行集群部署？
测试环境
1
2
3
4
5
6
7
8
9
10
| 测试环境：
Ubuntu 14.04 LTS x64
Tachyon：tachyon-0.7.1-bin.tar.gz
hostname IP role
spark-master： 192.168.108.20 master & worker
spark-slave1： 192.168.108.21 worker
spark-slave2： 192.168.108.22 worker
!默认情况全部操作在root下进行
|
Tachyon集群配置
准备工作
修改主机名
1
2
3
4
5
6
7
8
| /**
* 分别配置三台主机的主机名
* hostname：
* spark-master
* spark-slave1
* spark-slave2
*/
sudo vim /etc/hostname
|
修改hosts
1
2
3
4
5
6
7
8
9
10
| /**
* 分别配置修改三台主机的hosts
* 添加如下内容：
* 192.168.108.20 spark-master
* 192.168.108.21 spark-slave1
* 192.168.108.22 spark-slave2
* 如果发现hosts文件中有127.0.1.1或者除了127.0.0.1之外的其他内容，将他们删除
*
*/
sudo vim /etc/hosts
|
重启生效
spark-master 配置
请参考：Tachyon本地部署
分配ssh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| /**
* master主机中运行
* 切换至root
*/
su root
ssh-keygen -t rsa
/**
* 分发ssh至每台机子
*/
su root
scp ~/.ssh/id_rsa.pub root
scp ~/.ssh/id_rsa.pub root
scp ~/.ssh/id_rsa.pub root
|
授权ssh
1
2
3
4
5
6
7
8
| /**
* 分别在所有主机运行
*/
su root
cat /home/id_rsa.pub >> /root/.ssh/authorized_keys
cd ~/.ssh/
chmod 600 authorized_keys
|
修改ssh配置
1
2
3
4
5
6
7
8
9
| /**
* 分别在所有主机运行
*/
sudo vim /etc/ssh/sshd_config
注释：PermitRootLogin without-password
添加：PermitRootLogin yes
sudo service ssh restart
|
Tachyon环境变量
1
2
3
4
5
6
7
8
9
| /**
* 分别在所有主机运行，并添加如下内容
*/
sudo vim /etc/profile
export TACHYON_HOME=
export PATH=$TACHYON_HOME/bin:$PATH
source /etc/profile
|
Tachyon目录分发
1
2
3
4
5
6
7
8
9
10
11
12
| /**
* 复制master下的Tachyon目录到slave1、slave2下，并修改master配置
*/
sudo vim ./conf/workers
spark-master
spark-slave1
spark-slave2
sudo vim ./conf/slaves
spark-master
spark-slave1
spark-slave2
|
初始化Tachyon
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| /**
* master主机下运行
* 要用root权限运行
*/
sudo ./bin/tachyon format
Connecting to spark-master as root...
root@spark-master's password:
Formatting Tachyon Worker @ spark-master
Connection to spark-master closed.
Connecting to spark-slave1 as root...
Formatting Tachyon Worker @ spark-slave1
Connection to spark-slave1 closed.
Connecting to spark-slave2 as root...
Formatting Tachyon Worker @ spark-slave2
Connection to spark-slave2 closed.
Formatting Tachyon Master @ spark-master
|
启动Tachyon集群
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
| /**
* master主机下运行
* 要用root权限运行
*/
sudo ./bin/tachyon-start.sh all Mount
Killed 0 processes on spark-master
Killed 0 processes on spark-master
Connecting to spark-master as root...
root@spark-master's password:
Killed 0 processes on spark-master
Connection to spark-master closed.
Connecting to spark-slave1 as root...
Killed 0 processes on spark-slave1
Connection to spark-slave1 closed.
Connecting to spark-slave2 as root...
Killed 0 processes on spark-slave2
Connection to spark-slave2 closed.
Starting master @ spark-master
Connecting to spark-master as root...
root@spark-master's password:
Formatting RamFS: /mnt/ramdisk (512mb)
Starting worker @ spark-master
Connection to spark-master closed.
Connecting to spark-slave1 as root...
Formatting RamFS: /mnt/ramdisk (512mb)
Starting worker @ spark-slave1
Connection to spark-slave1 closed.
Connecting to spark-slave2 as root...
Formatting RamFS: /mnt/ramdisk (512mb)
Starting worker @ spark-slave2
Connection to spark-slave2 closed.
|
运行测试
查看每个主机Tachyon进程
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
* 每个主机切换到root下
* su root
*/
root@spark-master:
25598 TachyonWorker
25377 TachyonMaster
28559 Jps
root@spark-slave1:
4555 Jps
4518 TachyonWorker
root@spark-slave2:
4467 TachyonWorker
4502 Jps
|
查看Web UI
在master上用浏览器打开：hostname:19999 (spark-master:19999)
点击Works页面，会看到三台主机基本信息
