更多
更多

基于Hadoop的简单云盘

首先完成 hadoop完全分布式搭建

开发环境:

1
2
3
4
5
6
本机 macOS 10.12.5
虚拟机 CentOS 6.9 ×3
Hadoop 2.7.3
JDK 1.8
Tomcat 8.5.16
IDE Eclipse Java EE IDE for Web Developers, Oxygen Release (4.7.0)

安装Tomcat

下载Tomcat,我选择8.5.16的.tar.gz文件,解压并改名,最终路径为/usr/local/Tomcat

授予权限,sudo chmod 755 /usr/local/Tomcat/bin/*.sh

运行

1
2
cd /usr/local/Tomcat/bin
./startup.sh

在浏览器输入localhost:8080即可访问,关闭在终端输入./shutdown.sh


Eclipse配置Tomcat

  1. 打开设置,Server - Runtime Environments

  2. 右侧Add,选择对应Tomcat版本,next

  3. 路径填写Tomcat安装路径,我这里为/usr/local/Tomcat,并选择JRE

  4. Finish

  5. 点击IDE下方Serverscreate a new server, next

  6. Finish

  7. 双击创建的服务,Deploy path 填写Tomcat安装路径,即/usr/local/Tomcat,关闭保存


Eclipse创建Dynamic Web Project

  1. File-New-Other-Web-Dynamic Web Project


  2. 如图所示填写,创建一个名为TomcatTest的项目,之后一路next


  3. 选择下方服务,右键,选择Add and Remove,将左侧的TomcatTest添加到右边,完成。

  4. WebContent目录下新建index.html文件,写入

    1
    2
    3
    4
    5
    6
    7
    8
    9
    <html>
    <head>
    <title>Tomcat</title>
    </head>

    <body>
    <p>Test</p>
    </body>
    </html>
  5. 右键下方服务,Start


  6. 在浏览器输入localhost:8080/TomcatTest/index.html访问。

    本地tomcat测试成功


启动hadoop

1.启动3个虚拟机,

Update: 2017-9

事实证明,人确实容易犯懒,拖到现在😅。一旦放下等以后再整理,会花更多的时间和精力。

2.启动Hadoop

1
2
3
cd /usr/hadoop/sbin
./start-dfs.sh
./start-yarn.sh

50070端口:



在Eclipse里创建新项目

1.主页及子版块页面,结构类似,其框架及部分代码如下图

core-site.xmlhdfs-site.xml为hadoop配置文件,复制到项目下。

2.相关代码如下

  • FileInfo.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
package com.wh.entity;

public class FileInfo {

private String fileName;
private double fileSize;
private String createDate;

public FileInfo() {}

public FileInfo(String fileName, double fileSize, String createDate) {
this.fileName = fileName;
this.fileSize = fileSize;
this.createDate = createDate;
}

public String getFileName() {
return fileName;
}

public void setFileName(String fileName) {
this.fileName = fileName;
}

public double getFileSize() {
return fileSize;
}

public void setFileSize(double fileSize) {
this.fileSize = fileSize;
}

public String getCreateDate() {
return createDate;
}

public void setCreateDate(String createDate) {
this.createDate = createDate;
}

}


  • UserServlet.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
package com.wh.servlet;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.hadoop.fs.FileStatus;

import com.wh.entity.FileInfo;
import com.wh.util.HDFSUtil;

@WebServlet("/userServlet")
public class UserServlet extends HttpServlet {

@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {

}

@Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
//获取浏览器提交的数据
String userName = request.getParameter("userName");
String userPassword = request.getParameter("userPassword");

//把提交的数据与数据库中的用户信息进行
if("admin".equals(userName) && "123456".equals(userPassword)) {

FileStatus[] files = HDFSUtil.listFile();
List<FileInfo> listFile = new ArrayList<FileInfo>();
for(FileStatus fs : files) {
FileInfo file = new FileInfo(fs.getPath().getName(), fs.getLen(), "2017-07-01");
listFile.add(file);
}
request.setAttribute("fileInfos", listFile);

request.getRequestDispatcher("main.jsp").forward(request, response);

}else {
request.getRequestDispatcher("failure.html").forward(request, response);
}
}
}


  • FileUploadServlet.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
package com.wh.servlet;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;

import javax.servlet.ServletException;
import javax.servlet.annotation.MultipartConfig;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.Part;

import org.apache.hadoop.fs.FileStatus;

import com.wh.entity.FileInfo;
import com.wh.util.HDFSUtil;

@WebServlet("/fileUploadServlet")
@MultipartConfig
public class FileUploadServlet extends HttpServlet {

@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {

}

@Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
//设置中文编码
request.setCharacterEncoding("utf-8");
//获取客户端提交的文件
Part part = request.getPart("file");
//获取客户端提交的文件名
System.out.println(part.getSubmittedFileName());
//=========================文件上传==================================
InputStream in = part.getInputStream();
HDFSUtil.uploadFile(in, part.getSubmittedFileName());
//=========================文件列表==============================
FileStatus[] files = HDFSUtil.listFile();
List<FileInfo> listFile = new ArrayList<FileInfo>();
for(FileStatus fs : files) {
FileInfo file = new FileInfo(fs.getPath().getName(), fs.getLen(), "2017-07-01");
listFile.add(file);
}

request.setAttribute("fileInfos", listFile);
//页面跳转
request.getRequestDispatcher("main.jsp").forward(request, response);
}

}


  • FileDownloadServlet.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
package com.wh.servlet;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.URLEncoder;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.wh.util.HDFSUtil;

/**
* 文件下载
* @author Administrator
*
*/
@WebServlet("/fileDownLoadServlet")
public class FileDownloadServlet extends HttpServlet {

@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
request.setCharacterEncoding("UTF-8");
response.setCharacterEncoding("UTF-8");

String fileName = request.getParameter("fileName");
//解决中文乱码
response.setHeader("Content-Disposition", "attachment;filename="
+ new String(fileName.getBytes("GB18030"),"ISO8859-1"));

OutputStream out = response.getOutputStream();
//文件下载
HDFSUtil.downLoadFile(fileName, out);
}

@Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {

}
}


  • FileDeleteServlet.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
package com.wh.servlet;

import java.io.File;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.List;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.Part;

import org.apache.hadoop.fs.FileStatus;

import com.wh.entity.FileInfo;
import com.wh.util.HDFSUtil;

@WebServlet("/fileDeleteServlet")
public class FileDeleteServlet extends HttpServlet {
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
request.setCharacterEncoding("UTF-8");
response.setCharacterEncoding("UTF-8");

String fileName = request.getParameter("fileName");

HDFSUtil.deleteFile(fileName);
doPost(request, response);

}

@Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
FileStatus[] files = HDFSUtil.listFile();
List<FileInfo> listFile = new ArrayList<FileInfo>();
for(FileStatus fs : files) {
FileInfo file = new FileInfo(fs.getPath().getName(), fs.getLen(), "2017-07-01");
listFile.add(file);
}

request.setAttribute("fileInfos", listFile);
request.getRequestDispatcher("main.jsp").forward(request, response);
}

}


  • HDFSUtil.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
package com.wh.util;

import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

/**
* HDFS工具类
*
* @author Administrator
*
*/
public class HDFSUtil {

// 加载HDFS配置文件
private static Configuration conf = new Configuration();

// 上传
public static void uploadFile(InputStream in, String fileName)
throws IOException {
FileSystem hdfs = FileSystem.get(conf);
// 设置要上传文件的路径
Path uploadPath = new Path("/user/hadoop/" + fileName);
// 进行文件上传
FSDataOutputStream out = hdfs.create(uploadPath);
// 开始进行文件上传
IOUtils.copyBytes(in, out, conf);
// 关闭
IOUtils.closeStream(in);
IOUtils.closeStream(out);
}

// 下载
public static void downLoadFile(String fileName, OutputStream out) throws IOException {
FileSystem hdfs = FileSystem.get(conf);
// 定位要访问的文件
Path remoteFile = new Path("/user/hadoop/" + fileName);
// 与定位文件连接输入流管道
FSDataInputStream in = hdfs.open(remoteFile);
// 进行文件下载
IOUtils.copyBytes(in, out, conf);
// 关闭流
IOUtils.closeStream(in);
IOUtils.closeStream(out);
}

public static void deleteFile(String fileName) throws IOException {
FileSystem hdfs = FileSystem.get(conf);
Path remoteFile = new Path("/user/hadoop/" + fileName);
hdfs.delete(remoteFile,false);



}
// 文件列表展示
public static FileStatus[] listFile() throws IOException {
FileSystem hdfs = FileSystem.get(conf);
// 查看hdfs中某个文件目录
Path remotePath = new Path("/user/hadoop/");
// 获取目录下的文件列表
FileStatus[] files = hdfs.listStatus(remotePath);

return files;
}

// 删除
public static void deleteFile() throws IOException {
FileSystem hdfs = FileSystem.get(conf);
}
}


  • login.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>用户登录</title>
</head>
<body>
<div style="text-align:center;">
<h2>欢迎登录XX系统</h2>
<!-- action配置请求的服务器地址 -->
<!-- method表示HTTP请求的方式 -->
<form action="userServlet" method="post">
<p>
<label for="userName">用户名:</label>
<input type="text" id="userName" name="userName" placeholder="请输入用户名"/>
</p>

<p>
<label for="userPassword">密&nbsp;&nbsp;&nbsp;码:</label>
<input type="password" id="userPassword" name="userPassword" placeholder="请输入密码" />
</p>

<p><input type="submit" value="登录" /></p>
</form>
</div>
</body>
</html>


  • failure.html
1
2
3
4
5
6
7
8
9
10
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>登录失败</title>
</head>
<body>
用户名或密码错误,请<a href="login.html">重新登录</a>
</body>
</html>


  • main.jsp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Insert title here</title>
</head>
<body>
<center>
<table>
<tr>
<th>文件名</th>
<th>文件大小</th>
<th>创建时间</th>
<th>操作</th>
</tr>
<c:forEach var="file" items="${fileInfos}">
<tr>
<td>${file.fileName}</td>
<td>${file.fileSize}</td>
<td>${file.createDate}</td>
<td><a href="fileDownLoadServlet?fileName=${file.fileName}">下载</a>|<a href="fileDeleteServlet?fileName=${file.fileName}">删除</a></td>
</tr>
</c:forEach>
</table>


<!-- application/x-www-form-urlencoded -->
<form action="fileUploadServlet" method="post" enctype="multipart/form-data">
<p>
<label for="file">上传文件:</label>
<input type="file" id="file" name="file"/>
</p>

<p>
<input type="submit" value="上传文件" />
</p>
</form>
</center>
</body>
</html>


3.相关库

~/file/WebContent/WEB-INF/lib

注意从Tomcat中复制servlet-api.jar文件到lib

4.相关界面

  • 登录

  • 操作

  • 上传

  • 下载

  • 删除



总结

在项目制作过程中思路很重要,制作初期不能太纠结于个别效果,先按流程搭建好开发环境。

化繁为简,用最简单的方法,制作出理想的效果,实现全部的基础功能,实用第一,尽量美化。

拖了这么久,总算是基本完成了 _(:з」∠)_