Skip to content

V
Vietnamese text moderation
Commit 44f5233e authored by Cao Duc Anh's avatar Cao Duc Anh
Browse files
Options

add data

parent 107ec12c
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 236 additions and 78 deletions
README.md
View file @ 44f5233e
......@@ -16,6 +16,21 @@ Các file dữ liệu sẵn sàng cho huấn luyện ở định dạng *.csv g
- text: dữ liệu văn bản
- label: nhãn phân loại, chữ thường không dấu, phân cách từ bằng dấu "_"
Dữ liệu "khac" được crawl từ báo vietnamnet, báo nông nghiệp và một phần từ bộ dữ liệu "khieu_dam" mà không thuộc nhãn "khieu_dam". Code crawl: ./data/crawl_data.ipynb <br>
Dữ liệu "phan_dong" được crawl từ các fanpage phản động trên facebook. Code crawl: ./data/crawl_fb/ <br>
Dữ liệu "khieu_dam" được crawl từ truyensexcogiao.com và truyensex88.net. Code crawl: ./data/crawl_data.ipynb <br>
Dữ liệu "thu_ghet" sử dụng bộ dữ liệu ViHOS:
ViHOS chứa 26.476 khoảng thời gian được chú thích bởi con người trên 11.056 bình luận (5.360 bình luận có khoảng thời gian thù địch và xúc phạm, và 5.696 bình luận thì không)
```
@article{hoang2023vihos,
title={ViHOS: Hate Speech Spans Detection for Vietnamese},
author={Hoang, Phu Gia and Luu, Canh Duc and Tran, Khanh Quoc and Van Nguyen, Kiet and Nguyen, Ngan Luu-Thuy},
journal={arXiv preprint arXiv:2301.10186},
year={2023}
}
```
## Hướng dẫn triển khai với registry vivas
### 1. Pull docker images
Pull images with robot account:
......
data/crawl_data.ipynb
View file @ 44f5233e
This diff is collapsed.
data/crawl_fb/comments.csv 0 → 100644
View file @ 44f5233e
This source diff could not be displayed because it is too large. You can view the blob instead.
data/crawl_fb/config.yml 0 → 100644
View file @ 44f5233e
# change to list chars of your dataset or use default vietnamese chars
vocab: 'aAàÀảẢãÃáÁạẠăĂằẰẳẲẵẴắẮặẶâÂầẦẩẨẫẪấẤậẬbBcCdDđĐeEèÈẻẺẽẼéÉẹẸêÊềỀểỂễỄếẾệỆfFgGhHiIìÌỉỈĩĨíÍịỊjJkKlLmMnNoOòÒỏỎõÕóÓọỌôÔồỒổỔỗỖốỐộỘơƠờỜởỞỡỠớỚợỢpPqQrRsStTuUùÙủỦũŨúÚụỤưƯừỪửỬữỮứỨựỰvVwWxXyYỳỲỷỶỹỸýÝỵỴzZ0123456789!"#$%&''()*+,-./:;<=>?@[\]^_`{|}~ '
# cpu, cuda, cuda:0
device: cuda:0
data/crawl_fb/craw.py 0 → 100644
View file @ 44f5233e
This diff is collapsed.
data/crawl_fb/data_preprocess.csv 0 → 100644
View file @ 44f5233e
This source diff could not be displayed because it is too large. You can view the blob instead.
data/crawl_fb/post_crawl.csv 0 → 100644
View file @ 44f5233e
This diff is collapsed.
data/crawl_fb/post_ids.csv 0 → 100644
View file @ 44f5233e
851113453725330
851089323727743
851085110394831
851078773728798
851052437064765
851038853732790
851020597067949
851011447068864
850882573748418
850987603737915
850983610404981
850809940422348
850959567074052
850957543740921
850811710422171
850933937076615
850807877089221
850913160412026
850806707089338
850888490414493
850883173748358
850800170423325
850865387083470
850854523751223
850796513757024
850791173757558
850798670423475
850806970422645
850793423757333
850780797091929
850768517093157
850753127094696
850748540428488
850732000430142
850722003764475
850714307098578
850433927126616
850422310461111
850399313796744
850395567130452
850365587133450
850366253800050
850349620468380
850332020470140
850319207138088
850206053816070
850300567139952
850285613808114
850272587142750
850258347144174
850245223812153
850243450478997
850225377147471
850219730481369
850206760482666
850195130483829
850101860493156
850103593826316
850100927159916
850099830493359
850134563823219
850099073826768
850107563825919
850096687160340
850081570495185
850021330501209
850057893830886
850014180501924
850033763833299
850009600502382
850012177168791
850002943836381
849992003837475
849782760525066
849790030524339
849778517192157
849739837196025
849749503861725
849742017195807
849733017196707
849718487198160
849703050533037
849685970534745
849671250536217
849657130537629
849645047205504
849632373873438
849620993874576
849606527209356
849594587210550
849579993878676
849567520546590
849477477222261
849521747217834
849465843890091
849496720553670
849464100556932
849465517223457
849453290558013
849439810559361
849432303893445
849414357228573
849387970564545
849391047230904
849379530565389
849377830565559
data/crawl_fb/preprocess_data.ipynb 0 → 100644
View file @ 44f5233e
This source diff could not be displayed because it is too large. You can view the blob instead.
data/crawl_fb/tmp.ipynb 0 → 100644
View file @ 44f5233e
{
"cells": [
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1950\n"
]
}
],
"source": [
"import random \n",
"\n",
"sleep = random.randint(1000, 2000)\n",
"\n",
"print(sleep) "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "craw",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.8"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
data/list_data_sources.json
View file @ 44f5233e
[
{
"url": "https://truyensexcogiao.com/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/chinh-tri",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/2/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/thoi-su",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/3/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/kinh-doanh",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/4/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/the-thao",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/5/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/the-gioi",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/6/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/giao-duc",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/7/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/giai-tri",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/8/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/van-hoa",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/9/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/doi-song",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/10/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/suc-khoe",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/11/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/thong-tin-truyen-thong",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/12/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/phap-luat",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/13/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/oto-xe-may",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/14/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/bat-dong-san",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensexcogiao.com/page/15/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/du-lich",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensex88.net/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/ban-doc",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensex88.net/page/2/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/nong-thon-moi",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensex88.net/page/3/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/hop-tac/cong-nghiep-ho-tro",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensex88.net/page/4/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/dan-toc-ton-giao",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensex88.net/page/5/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/dan-toc-thieu-so-va-mien-nui",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
},
{
"url": "https://truyensex88.net/page/6/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
},
{
"url": "https://truyensex88.net/page/7/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
},
{
"url": "https://truyensex88.net/page/8/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
},
{
"url": "https://truyensex88.net/page/9/",
"div_class": "noibat",
"child_div_class": "ndtruyen"
"url": "https://vietnamnet.vn/thi-truong-tieu-dung",
"div_class": "horizontalPost__avt",
"child_div_class": "main-content"
}
]
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment