storm集群和单点安装
< 返回列表时间: 2016-02-06来源:OSCHINA
Storm实时分析平台的起源,Storm分布式集群实施
流式处理场景:算法交易
storm组件
1.nimbus集群中心,控制节点
2.supervisor nodes taskernode
3.the zookeeper cluster 交换信息
storm数据模型:由许多记录组成,送入
环境准备
1.关闭防火墙
2.修改hosts
192.168.88.197 user5.hadoop.com
192.168.88.198 user6.hadoop.com
192.168.88.196 user7.hadoop.com
wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz
tar -zxvf zookeeper-3.3.6.tar.gz
cd zookeeper-3.3.6
mkdir data
cd conf/
cp -rp zoo_sample.cfg zoo.cfg
dataDir=/root/soft/zookeeper-3.3.6/data
前一个端口选举,后一个端口zookeeper进行通讯
server.1=user5.hadoop.com:2888:3888
server.2=user6.hadoop.com:2888:3888
server.3=user7.hadoop.com:2888:3888
cd ~
scp -rp zookeeper-3.3.6/ root@ip:/home/grid
[root @localhost data]# cd /root/soft/zookeeper-3.3.6/data

创建myid
在dataDir(/home/grid/zookeeper-3.4.6/data)中创建一个myid
因为server.1=user5.hadoop.com:2888:3888指定的是1,所以在user5.hadoop.com的
机器上echo “1” >myid
其余两台机器配置,user6.hadoop.com myid=2 user7.hadoop.com myid=3

/root/soft/zookeeper-3.3.6/bin
[root @localhost bin]#
[root @localhost bin]# ./zkServer.sh start
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root @localhost bin]# ./zkServer.sh status
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: leader
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@localhost bin]# ./zkServer.sh status
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: follower
[root@localhost bin]# ./zkServer.sh start
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@localhost bin]# ./zkServer.sh status
JMX enabled by default
Using config: /root/soft/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: follower

安装python
https://www.python.org/downloads/ftp/python2.6.6
python
[root@localhost Desktop]# tar -zxf Python-2.7.2.tgz
[root@localhost soft]# cd Python-2.7.2
./configure --prefix=/usr/local/python27
make
make install
[root@localhost Python-2.7.2]# ls /usr/local/python27/ -al
total 28
drwxr-xr-x 6 root root 4096 Feb 5 23:29 .
drwxr-xr-x 12 root root 4096 Feb 5 23:28 ..
drwxr-xr-x 2 root root 4096 Feb 5 23:29 bin
drwxr-xr-x 3 root root 4096 Feb 5 23:29 include
drwxr-xr-x 4 root root 4096 Feb 5 23:29 lib
drwxr-xr-x 3 root root 4096 Feb 5 23:29 share
[root@localhost Python-2.7.2]# mv /usr/bin/python /usr/bin/python_old
[root@localhost Python-2.7.2]# ln -s /usr/local/python27/bin/python /usr/bin/
[root@localhost Python-2.7.2]# python
Python 2.7.2 (default, Feb 5 2016, 23:28:47)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
[root@localhost Python-2.7.2]# vi /usr/bin/yum
#!/usr/bin/python2.4

0.9之前的安装步骤
同样安装JDK和python,zookeeper集群
2.和前面一样,只是要多安装storm的两个依赖库,zeroMQ和JZMQ
1.搭建zookeper集群
2.安装storm依赖库【JDK和python,zeroMQ,JZMQ】
3.下载并且解压storm发布版本
4.修改storm.yaml文件
5.启动storm后台进程

[root@localhost soft]# wget http://mirror.bit.edu.cn/apache/storm/apache-storm-0.9.1-incubating/apache-storm-0.9.1-incubating.tar.gz
[root@localhost soft]# tar -zxf apache-storm-0.9.1-incubating.tar.gz
t@localhost soft]# tar -zxf apache-storm-0.9.1-incubating.tar.gz
@localhost soft]# mv apache-storm-0.9.1-incubating storm-0.9.1
[root@localhost soft]# cd storm-0.9.1
[root@localhost storm-0.9.1]# cd conf
[root@localhost conf]# ls
[root@localhost soft]# cd storm-0.9.1
[root@localhost storm-0.9.1]# cd conf
[root@localhost conf]# ls
storm.yaml
[root@localhost conf]# vi storm.yaml

storm.zookeeper.servers:
- "192.168.88.197"
- "192.168.88.198"
- "192.168.88.196"
#
nimbus.host: "192.168.88.197"
storm.local.dir:"/root/soft/storm-0.9.1/data"
supervisor.slots.ports:
-6700
-6701
-6702
-6703



/root/soft/storm-0.9.1
[root@localhost soft]# scp -rp storm-0.9.1/ root@user6.hadoop.com:~/soft
[root@localhost soft]# scp -rp storm-0.9.1/ root@user7.hadoop.com:~/soft
.BASH_PROFILE
STORM_HOME=/root/soft/storm-0.9.1
PATH=$PATH:$HOME/bin:$STORM_HOME/bin
export STORM_HOME最下面

[root@localhost storm-0.9.1]# source /etc/profile
[root@localhost storm-0.9.1]# echo $PATH
/root/soft/storm-0.9.1/bin
[root@localhost ~]# jps确保三台都能够正常启动
5215 QuorumPeerMain
23327 Jps

[root@localhost conf]# storm nimbus &在nimbus节点
[root@localhost conf]# jps
5536 nimbus
5609 Jps
5431 QuorumPeerMain
[root@user6 ~]# storm supervisor &
[root@user6 ~]# jps
\5165 QuorumPeerMain
24473 Jps
24354 supervisor


[root@localhost conf]# jps
5536 nimbus
5609 Jps
5431 QuorumPeerMain
[root@localhost conf]# storm logviewer &
[root@localhost conf]# storm ui &网址8080端口
[root@localhost conf]# jps
5536 nimbus
5841 logviewer
6013 core
5431 QuorumPeerMain
6074 Jps


[root@user6 ~]# jps
5165 QuorumPeerMain
24678 Jps
24354 supervisor


安装git
yum install git或者 http://distfiles.macports.org/git-core/git-1.8.4.2.tar.gz


wget http://mirrors.ustc.edu.cn/fedora/epel/epel-release-latest-5.noarch.rpm



安装maven
wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
解压配置环境变量
[root@localhost soft]# git clone git://github.com/nathanmarz/storm-starter.git
[root@localhost soft]# cd storm-starter
[root@localhost storm-starter]# ls
LICENSE multilang README.markdown storm-starter.iml
m2-pom.xml project.clj src test
[root@localhost storm-starter]# mv m2-pom.xml pom.xml
[root@localhost storm-starter]# vi pom.xml
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.1-incubating</version>
<!-- keep storm out of the jar-with-dependencies -->
<scope>provided</scope>
</dependency>

[root@localhost storm-starter]# mvn compile exec:java -Dstorm.topology=storm.starter.WordCountTopology
mvn package
storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.RollingTopwords
远程模式
storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.ExclamationTopology ExclamationTopology
storm kill ExclamationTopology
安装ZeroMQ
安装依赖包
yum install gcc-c++ libuuid-devel
对于libuuid-devel
先安装ncurses-5.9.tar
在下载ftp://ftp.kernel.org/pub/linux/utils/util-linux/v2.21/util-linux-2.21.1.tar.gz
编译安装
下载http://download.zeromq.org/zeromq-2.1.7.tar.gz
编译安装JZMQ
git clone git://github.com/nathanmarz/jzmq.git
cd jzmq
[root@localhost jzmq]# ./autogen.sh
[root@localhost jzmq]# ./configure
make
make install
下载并解压storm第三台
yum install unzip
wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip
ocalhost soft]# unzip storm-0.8.1.zip
[root@localhost soft]# cd storm-0.8.1
cd storm-0.8.1
修改conf/storm.yaml文件在配置的每一项前加空格,冒号后也要加空格
storm.zookeeper.servers:
- "192.168.88.197"
- "192.168.88.198"
- "192.168.88.199"
#
nimbus.host: "192.168.88.199"
storm.local.dir: "/root/soft/storm-0.8.1/data"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
[root@localhost conf]# jps
5215 QuorumPeerMain
13883 Jps
修改环境变量

[root@localhost ~]# storm nimbus &
[1] 13974
[root@localhost ~]# jps
[root@localhost ~]# jps
14127 Jps
13974 nimbus
5215 QuorumPeerMain
14037 logviewer
14059 core
mvn compile exec:java -Dstorm.topology=storm.starter.WordCountTopology
[root@localhost storm-starter]# mvn package
[root@localhost storm-starter]# cd target
[root@localhost target]# storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.RollingTopWords本地

[root@localhost target]# storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.ExclamationTopology ExclamationTopology集群方式




热门排行