阅读量:5
目录
一、前期准备
1.1 部署 docker
# 安装yum-config-manager配置工具 yum -y install yum-utils # 建议使用阿里云yum源：（推荐） #yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 安装docker-ce版本 yum install -y docker-ce # 启动并开机启动 systemctl enable --now docker docker --version docker info
1.2 部署 docker-compose
curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose docker-compose --version
创建网络
docker network create hadoop-network1.3 版本说明
组件 | 版本 |
操作系统 | Centos7.6/Ubuntu20.04.5 |
ambari | 2.7.4 |
HDP | 3.1.4 |
HDP-GPL | 3.1.4 |
HDP-UTILS | 1.1.0.22 |
JDK | jdk-8u162-linux-x64 |
如下安装只做测试。
二 、镜像构建启动
2.1 系统镜像构建
# 拉取 Centos 镜像 FROM centos:7.6.1810 # 作者信息 LABEL kangll # 安装基本工具 RUN yum update -y && yum install -y \ net-tools \ vim \ wget \ curl \ lrzsz \ nc \ openssl \ sed \ telnet \ openssh-server \ httpd \ chronyd \ zip unzip \ openssh-clients \ expect \ krb5-server \ krb5-libs \ krb5-workstation \ && yum clean all # JDK RUN mkdir -p /usr/java ADD ./config/jdk-8u162-linux-x64.tar.gz /usr/java/ ENV JAVA_HOME=/usr/java/jdk1.8.0_162 ENV CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar ENV PATH=$JAVA_HOME/bin:$PATH # 添加一个名为myuser的用户 RUN useradd -m winner_spark # 设置myuser用户的密码，这里设置为mypassword RUN echo 'winner_spark:123456' | chpasswd # 设置myuser用户为sudoer RUN echo 'winner_spark ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers # SSH RUN sed -i '/^HostKey/'d /etc/ssh/sshd_config RUN echo 'HostKey /etc/ssh/ssh_host_rsa_key' >> /etc/ssh/sshd_config RUN mkdir /var/run/sshd/ RUN echo 'UseDNS no' >> /etc/ssh/sshd_config RUN sed -i -e '/pam_loginuid.so/d' /etc/pam.d/sshd COPY ./config/hosts /etc/ #set default password RUN echo 'root:123456' | chpasswd RUN /usr/bin/ssh-keygen -A COPY ./config/krb5.conf /etc #声明22端口 EXPOSE 22 RUN mkdir -p /opt && mkdir -p /var/log/supervisord RUN echo '/usr/sbin/sshd -D' >> /opt/run.sh RUN chmod +x /opt/run.sh CMD ["/opt/run.sh"] # WORKDIR /opt/系统镜像构建
docker build -f dockerfile-centos -t centos-jdk:1.0 .2.2 安装包源镜像构建
dockerfile-httpd
FROM centos-jdk:1.0 # centos7.6 JDK1.8 LABEL kangll RUN yum -y install httpd RUN echo "crushlinux" >/var/www/html/index.html RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime # 添加yum 源 COPY ./repo/ambari.repo /etc/yum.repos.d/ COPY ./repo/hdp.repo /etc/yum.repos.d/ COPY ./repo/hdp.gpl.repo /etc/yum.repos.d/ RUN mkdir -p /hadoop/hdp ADD ./HDP/ambari.tar.gz /var/www/html ADD ./HDP/HDP-GPL.tar.gz /var/www/html ADD ./HDP/HDP.tar.gz /var/www/html ADD ./HDP/HDP-UTILS.tar.gz /var/www/html EXPOSE 80 CMD ["httpd","-DFOREGROUND"] 源镜像构建
docker build -t ambari:2.7.4 -f dockerfile-ambari . 
2.3 kdc镜像构建
dockerfile-kerberos
FROM centos:7.6.1810 RUN yum install -y krb5-server krb5-libs krb5-auth-dialog krb5-workstation RUN yum install -y net-tools telnet COPY ./config/krb5.conf /etc COPY ./config/kdc.conf /var/kerberos/krb5kdc/ COPY ./config/kadm5.acl /var/kerberos/krb5kdc/ CMD ["/usr/sbin/init"]服务启动
# 进入容器 启动服务 Systemctl restart krb5kdc Systemctl restart kadmindocker-compose-base.yml 启动基础环境
version: '3' services: mysql: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7 restart: always networks: - hadoop-network container_name: mysql hostname: mysql-test environment: MYSQL_ROOT_PASSWORD: Winner001 TZ: Asia/Shanghai ports: - 3306:3306 volumes: - /data:/var/lib/mysql - ./conf/my.cnf:/etc/mysql/my.cnf command: --max_connections=1000 --character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci --default-authentication-plugin=mysql_native_password healthcheck: test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"] interval: 10s timeout: 10s httpd: container_name: httpd hostname: httpd image: httpd:1.0 networks: - hadoop-network ports: - "80:80" privileged: true restart: always volumes: - "/etc/hosts:/etc/hosts" kdc: image: kerberos:1.0 restart: always networks: - hadoop-network container_name: kerberos hostname: kerberos.winner ports: - 88:88 - 749:749 - 750:750 volumes: - /etc/hosts:/etc/hosts - /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc - /hadoop/kerberos/keytabs:/etc/security/keytabs command: - /bin/sh - -c - | /opt/apache/kerberosStart.sh networks: hadoop-network: external: name: hadoop-network Ambari启动
# 启动基础环境 docker-compose -f docker-compose-base.yaml up -d # 启动ambari , 在此容器中安装集群 docker-compose -f docker-compose-ambari.yaml up -d 2.4 集群安装
repo地址
节点和秘钥
agent安装
服务安装
由于是单机直接下一步
安装组件
输入密码
数据目录确认
用户确认下一步
服务参数确定
发布
安装完成
2.5 容器导出为像
docker ps -a docker commit 容器ID ambari-1.0.tar.gz三、Ubuntu环境安装测试
3.1 环境准备
我们准备Ubuntu 环境，在Ubuntu环境安装Ambari单机，相当于在Centos服务器安装的docker化的Ambari 导出为镜像 然后在Ubutntu 环境尝试启动。
准备Ubuntu 环境，版本：Ubuntu 20.04.5 LTS
将镜像 ambari-1.0.tar.gz 加载到Ubuntu 环境
docker load -i ambari-1.0.tar.gzMySQL元数据库备份还原（Ambari、hive、ranger）
HDFS、kerberos 备份的数据拷贝到Ubuntu服务器并保持数据文件路径不变。
将在centos 安装完成的集群备份元数据库写入到新的库中 , 也就是Ubuntu环境要使用的元数据库。
# 备份的元数据写入新的数据库中 mysql -hmysql-test -uroot -pWinner001 ambari < ambari.sql mysql -hmysql-test -uroot -pWinner001 ranger < ranger.sql mysql -hmysql-test -uroot -pWinner001 hive < hive.sql 备份的kerberos和安装包解压
# 解压到 /usr 路径下，保存的是集群安装的数据 hdp.tar.gz # 在根路径解压保留的kerberos 和HDFS 初始化的元数据 hadoop.tar.gz 
基础环境启动docker-compose-base.yaml
version: '3' services: mysql: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7 restart: always networks: - hadoop-network container_name: mysql hostname: mysql-test environment: MYSQL_ROOT_PASSWORD: Winner001 TZ: Asia/Shanghai ports: - 3306:3306 volumes: - /data:/var/lib/mysql - ./conf/my.cnf:/etc/mysql/my.cnf command: --max_connections=1000 --character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci --default-authentication-plugin=mysql_native_password healthcheck: test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"] interval: 10s timeout: 10s httpd: container_name: httpd hostname: httpd privileged: true image: httpd:1.0 networks: - hadoop-network ports: - "80:80" privileged: true restart: always volumes: - "/etc/hosts:/etc/hosts" kdc: image: kerberos:1.0 restart: always privileged: true networks: - hadoop-network container_name: kerberos hostname: kerberos.winner ports: - 888:88 - 8749:749 - 8750:750 volumes: - /etc/hosts:/etc/hosts - /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc - /hadoop/kerberos/keytabs:/etc/security/keytabs networks: hadoop-network: external: name: hadoop-network 启动
# 启动 docker-compose -f docker-compose-base.yaml up -d # 停止 docker-compose -f docker-compose-base.yaml down 3.2 集群容器启动
镜像
启动容器
version: '3' services: ambari: image: ambari:2.7.4 # 修改为新的容器 container_name: ambari hostname: ambari-server restart: always privileged: true networks: - hadoop-network ports: - 8080:8080 - 8088:8088 - 50070:50070 - 50075:50075 - 8188:8188 - 8042:8042 - 10000:10000 - 9083:9083 - 6080:6080 - 16010:16010 volumes: - /etc/hosts:/etc/hosts - /usr/hdp:/usr/hdp - /hadoop/hdfs:/hadoop/hdfs - /hadoop/etc/hadoop/:/etc/hadoop/ - /hadoop/etc/hbase/:/etc/hbase/ - /hadoop/etc/hive/:/etc/hive/ - /hadoop/etc/livy2/:/etc/livy2/ - /hadoop/etc/spark2/:/etc/spark2/ - /hadoop/etc/tez/:/etc/tez/ - /hadoop/etc/zookeeper/:/etc/zookeeper/ - /hadoop/yarn:/hadoop/yarn - /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc - /hadoop/kerberos/keytabs:/etc/security/keytabs command: - /bin/sh - -c - | /opt/apache/bootstrap.sh networks: hadoop-network: external: name: hadoop-network 启动
docker-compose -f docker-compose-ambari.yaml up -d
下次安装直接加载镜像，然后执行yaml文件即可
docker load -i ambari-1.0.tar.gz docker load -i kerberos.tar.gz docker load -i mysql.tar.gz docker load -i HDP-REPO.tar.gz 脚本
bootstrap.sh
#!/bin/bash set -x ambari_db=ambari myurl=mysql-test myuser=root mypwd=Winner001 myport=3306 mydb=ipvacloud java_home=/usr/java/jdk1.8.0_162 mysql_driver=/usr/share/java/mysql-connector-java.jar BASEDIR=$(cd "$(dirname "$0")"; pwd) wait_for() { echo Waiting for $1 to listen on $2... while ! nc -z $1 $2; do echo waiting...; sleep 1s; done } setup_ambari() { # 强制替换文件 cp -f /tmp/ambari.properties /etc/ambari-server/conf/ cp -f /tmp/password.dat /etc/ambari-server/conf/ cp -f /tmp/ambari-agent.ini /etc/ambari-agent/conf/ } # wait for mysql and httpd #wait_for windp-aio 3306 #wait_for httpd 80 # 开始ambari安装 yum -y install ambari-server yum -y install ambari-agent re=`mysql -h${myurl} -u${myuser} -p${mypwd} -e "show databases"|grep ambari|wc -l` if [ $re -eq 0 ];then # sql 初始化 mysql -h${myurl} -u${myuser} -p${mypwd} < /tmp/init_db.sql mysql -h${myurl} -u${myuser} -p${mypwd} ambari < /tmp/Ambari-DDL-MySQL-CREATE.sql fi # 配置 setup_ambari # 启动ambari服务 /usr/sbin/ambari-server start /usr/sbin/ambari-agent start /usr/sbin/sshd -D tail -f /var/log/ambari-server/ambari-server.log 参考文章：https://www.cnblogs.com/liugp/p/17489616.html
Ubuntu 环境安装docker: 【云计算】Ubuntu多种安装docker方式_ubuntu安装docker-CSDN博客
