利用贝叶斯分类器进行文本挖掘---笔记-白红宇

利用贝叶斯分类器进行文本挖掘---笔记

阅读量：5981 次

发布时间：2019-06-20

本文共 5023 字，大约阅读时间需要 16 分钟。

1. 调用庖丁分词器，分词

grid@server01:~/data$ hadoop jar mrtokenize.jar tokenize.TokenizeDriver /home/grid/data/lesson8 /home/grid/output/sportwords

14/08/31 21:59:33 INFO input.FileInputFormat: Total input paths to process : 10205

.....

14/08/31 22:05:25 INFO mapred.JobClient: Map output records=10205

共处理：10205个文件，耗时 6 分钟

分词后的结果：

badminton 尤伯伯杯中国泰国挺进决赛王仪仪涵机敏回球月日中国中国队选手王仪仪涵比赛中回回球她最最终战胜泰国队选手手拉差诺当日武汉体育中心中心体体育体育馆举行年尤尤伯伯杯羽毛羽毛球球赛决赛半决赛中中国中国队总比比分战胜泰国队晋级决赛新华新华社记者孟永永民民摄关注体育了解更多体坛资讯

badminton 尤伯伯杯中国泰国挺进决赛王仪仪涵比赛中发球月日中国中国队选手王仪仪涵比赛中发球她最最终战胜泰国队选手手拉差诺当日武汉体育中心中心体体育体育馆举行年尤尤伯伯杯羽毛羽毛球球赛决赛半决赛中中国中国队总比比分战胜泰国队晋级决赛新华新华社记者孟永永民民摄关注体育了解更多体坛资讯

2. load 分词集

grunt> processed = load'/home/grid/output/sportwords' as (category:chararray, doc:chararray);

3. 随机 20% 生成测试集

grunt> test = sample processed 0.2;

4. 生成训练集

grunt> jnt =join processed by (category,doc) left outer, test by (category, doc);

grunt> filt_test = filter jnt by test::category is null;

grunt> train = foreach filt_test generate processed::category as category, processed::doc as doc;

5. 导出测试集和训练集

grunt> store test into '/home/grid/data/lesson8.2/test'；

6. 统计测试集分类

grunt> test_ct= foreach(group test by category) generate group, COUNT(test.category);

grunt> dump test_ct;

result：

(f1,196)

(golf,206)

(swim,170)

(tennis,187)

(football,209)

(pingpong,220)

(badminton,212)

(billiards,236)

(basketball,201)

(volleyball,204)

7. 统计训练集

grunt> train_ct= foreach(group train by category) generate group, COUNT(train.category);

grunt> dump train_ct;

8. 利用朴素贝叶斯分类器，来训练模型

grid@server01:~/data$ mahout trainclassifier \

> -i /home/grid/data/lesson8.2/train \

> -o /home/grid/output/model-bayes8.2 \

> -type bayes \

> -ng 1 \

> -source hdfs

result:

14/08/31 23:31:15 INFO mapred.JobClient: Map output records=228211

14/08/31 23:31:15 INFO common.HadoopUtil: Deleting /home/grid/output/model-bayes8.2/trainer-docCount

14/08/31 23:31:15 INFO common.HadoopUtil: Deleting /home/grid/output/model-bayes8.2/trainer-termDocCount

14/08/31 23:31:15 INFO common.HadoopUtil: Deleting /home/grid/output/model-bayes8.2/trainer-featureCount

14/08/31 23:31:15 INFO common.HadoopUtil: Deleting /home/grid/output/model-bayes8.2/trainer-wordFreq

14/08/31 23:31:15 INFO common.HadoopUtil: Deleting /home/grid/output/model-bayes8.2/trainer-tfIdf/trainer-vocabCount

14/08/31 23:31:15 INFO driver.MahoutDriver: Program took 1265118 ms (Minutes: 21.0853)

9. 测试模型

grid@server01:~/data$ mahout testclassifier \

> -d /home/grid/data/lesson8.2/test \

> -m /home/grid/output/model-bayes8.2 \

> -type bayes \

> -ng 1 \

> -source hdfs \

> -method mapreduce

14/08/31 23:45:52 INFO bayes.BayesClassifierDriver: =======================================================Confusion Matrix-------------------------------------------------------a       b       c       d       e       f       g       h       i       j       <--Classified as190     0       1       1       0       3       0       0       0       0        |  195         a     = basketball0       249     0       0       0       0       1       0       0       0        |  250         b     = billiards0       0       198     0       0       1       0       0       0       0        |  199         c     = badminton0       0       0       224     0       0       0       0       0       0        |  224         d     = football0       0       0       0       190     0       0       0       0       0        |  190         e     = volleyball0       0       0       0       0       181     0       0       0       0        |  181         f     = swim0       1       0       0       0       0       204     0       0       0        |  205         g     = pingpong1       0       0       0       0       0       0       193     0       0        |  194         h     = golf0       0       0       0       0       0       0       0       196     0        |  196         i     = f10       0       0       0       0       0       1       0       0       206      |  207         j     = tennis

10. 用户浏览记录分词，同1

grid@server01:~/data$ hadoop jar mrtokenize.jar tokenize.TokenizeDriver /home/grid/data/les8-usersport /home/grid/lesson8/output/userwords

11. 利用sport 生成的模型对用户浏览内容进行分类

grid@server01:~/data$ hadoop jar mrclassify.jar classifier.ClassifierDriver \

> /home/grid/lesson8/output/userwords \

> /home/grid/lesson8/output/classify \

> /home/grid/output/model-bayes8.2 \

> bayes

result:

grid@server01:~/data$ hadoop fs -cat /home/grid/lesson8/output/classify/part-r-00000 | head -20

Warning: $HADOOP_HOME is deprecated.

10511838|badminton|7