diff --git a/AUTHORS b/AUTHORS new file mode 100644 index 0000000..9d7ae8d --- /dev/null +++ b/AUTHORS @@ -0,0 +1,3 @@ +The following authors have created the source code of "toloka-pachyderm" published and distributed by YANDEX LLC as the owner: + +Daniil Fedulov mr-fedulow@yandex-team.ru diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..85f6d25 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,35 @@ +# Notice to external contributors + + +## General info + +Hello! In order for us (YANDEX LLC) to accept patches and other contributions from you, you will have to adopt our Yandex Contributor License Agreement (the “**CLA**”). The current version of the CLA can be found here: +1) https://yandex.ru/legal/cla/?lang=en (in English) and +2) https://yandex.ru/legal/cla/?lang=ru (in Russian). + +By adopting the CLA, you state the following: + +* You obviously wish and are willingly licensing your contributions to us for our open source projects under the terms of the CLA, +* You have read the terms and conditions of the CLA and agree with them in full, +* You are legally able to provide and license your contributions as stated, +* We may use your contributions for our open source projects and for any other project too, +* We rely on your assurances concerning the rights of third parties in relation to your contributions. + +If you agree with these principles, please read and adopt our CLA. By providing us your contributions, you hereby declare that you have already read and adopt our CLA, and we may freely merge your contributions with our corresponding open source project and use it further in accordance with terms and conditions of the CLA. + +## Provide contributions + +If you have already adopted terms and conditions of the CLA, you are able to provide your contributions. When you submit your first pull request, please add the following information into it: + +``` +I hereby agree to the terms of the CLA available at: [link]. +``` + +Replace the bracketed text as follows: +* [link] is the link to the current version of the CLA: https://yandex.ru/legal/cla/?lang=en (in English) or https://yandex.ru/legal/cla/?lang=ru (in Russian). + +It is enough to provide this notification only once. + +## Other questions + +If you have any questions, please mail us at opensource@yandex-team.ru. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..ee9684e --- /dev/null +++ b/LICENSE @@ -0,0 +1,13 @@ +Copyright 2022 YANDEX LLC + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 120000 index 0000000..89113b7 --- /dev/null +++ b/README.md @@ -0,0 +1 @@ +example/README.md \ No newline at end of file diff --git a/example/README.md b/example/README.md new file mode 100644 index 0000000..13fad5f --- /dev/null +++ b/example/README.md @@ -0,0 +1,260 @@ +# Pachyderm example: Use Toloka to enrich data for clickbait detection + +We are going to use [Pachyderm](https://pachyderm.com/) to create a project, run news headlines annotation tasks in [Toloka](https://toloka.ai/) project, aggregate data +and train a model. + +So, there is to be 5 Pachyderm pipeines: + +1. Project creation +2. Training creation (including tasks upload) +3. Pool creation (including tasks upload) +4. Starting a pool and waiting for pool to complete +5. Assignments aggregation + +We also will run 2 pipeline as examples of further data processing: +* Datasets concatenation +* Training and testing Random Forest classifier for clickbait headings detection + +To set up Pachyderm locally please follow [Pachyderm Local Installation documentation](https://docs.pachyderm.com/latest/getting_started/local_installation/). + +Now, let's create these pipelines: + +## Setup + +1. Before creating pipelines, we need to create a secret with an API key. First things first, we need to create a secret + in kubernetes: + +```bash +kubectl create secret generic toloka-api --from-literal=token= --dry-run=client --output=json > toloka-api-key.json +``` + +2. Then we push this kubernetes token to Pachyderm and check it was added correctly + +```bash +pachctl create secret -f toloka-api-key.json +pachctl list secret +``` + +3. In the end of this step we are going to create docker for pipelines: + +```bash +docker build -f src/Dockerfile -t toloka_pachyderm:latest . +``` + +4. We also have to add our train and test datasets to pachyderm repo:| + +```bash +pachctl create repo clickbait_data +pachctl put file clickbait_data@master:train.csv -f ./data/train.csv +pachctl put file clickbait_data@master:test.csv -f ./data/test.csv + +``` + +## Project creation + +You may run project creation script: + +```bash +./create_toloka_project.sh +``` + +or create project step-by-step: + +1. Init pachyderm repo for project config + +```bash +pachctl create repo toloka_project_config +``` + +2. Put project config provided + +```bash +pachctl put file toloka_project_config@master:project.json -f ./configs/project.json +``` + +3. Check if data have been put to repository correctly + +```bash +pachctl list file toloka_project_config@master +``` + +4. Now let's create a pipeline + +```bash +pachctl create pipeline -f toloka_create_project.json +``` + +5. When pipeline is created, a new job is started. Let's check if the pipeline's job completed successfully + +```bash +pachctl list job -p toloka_create_project +``` + +## Training creation + +Training is needed for Tolokers to understand how to annotate your data properly. + +You may run training creation script: + +```bash +./create_toloka_training.sh +``` + +or initialize training step-by-step: + +1. Init pachyderm repo for training config + +```bash +pachctl create repo toloka_training_config +``` + +2. Put training config provided + +```bash +pachctl put file toloka_training_config@master:training.json -f ./configs/training.json +``` + +3. Check if data have been put to repository correctly + +```bash +pachctl list file toloka_training_config@master +``` + +4. Now let's create a pipeline + +```bash +pachctl create pipeline -f toloka_create_training.json +``` + +5. Let's check if the pipeline's job completed successfully + +```bash +pachctl list job -p toloka_create_training +``` + +6. But our training contains no tasks for Tolokers to study how to annotate your data. To fix this, we are going to + create a repo for training data and upload them: +```bash +pachctl create repo toloka_training_tasks +pachctl put file toloka_training_tasks@master:training_tasks.csv -f ./data/training_tasks.csv +pachctl create pipeline -f toloka_create_training_tasks.json +pachctl list job -p toloka_create_training_tasks +``` + +## Create pool + +Now let's create a pool – a set of paid tasks sent out for completion at the same time. + +You may run pool creation script: + +```bash +./create_toloka_pool.sh +``` + +or initialize pool step-by-step: + +1. Init pachyderm repo for pool config + +```bash +pachctl create repo toloka_pool +``` + +2. Put pool config and task data provided (we will work with tasks in the next stage) + +```bash +pachctl put file toloka_pool@master:pool.json -f ./configs/pool.json +pachctl put file toloka_pool@master:control_tasks.csv -f ./data/control_tasks.csv +pachctl put file toloka_pool@master:pool_tasks.csv -f ./data/pool_tasks.csv +``` + +3. Check if data have been put to repository correctly + +```bash +pachctl list file toloka_pool@master +``` + +4. Now let's create a pipeline + +```bash +pachctl create pipeline -f toloka_create_pool.json +``` + +5. Let's check if the pipeline's job completed successfully + +```bash +pachctl list job -p toloka_create_pool +``` + +## Adding tasks to pool + +Each item we need to get annotated with Toloka called Task. There are three types for Tasks: training Task, control Task and (simple) Task. +You've already worked with training Tasks when created Training. These type of tasks has a correct answer and a hint for Tolokers in description. +Control Tasks are used to check Tolokers labelling quality: they have a correct answer in description and Toloka checks whether Toloker's response matches the correct answer. If many control tasks were not labelled by Toloker correctly, they may be banned. + +You may run pool tasks creation script: + +```bash +./create_toloka_pool_tasks.sh +``` + +or initialize pool step-by-step: + +1. Let's run a pipeline to upload these control tasks to Toloka: +```bash +pachctl create pipeline -f toloka_create_control_tasks.json +pachctl list job -p toloka_create_control_tasks +``` +2. We also need to run a pipeline to upload the tasks we need to get annotations of to Toloka: +```bash +pachctl create pipeline -f toloka_create_pool_tasks.json +pachctl list job -p toloka_create_pool_tasks +``` + +## Wait for pool to complete + +In the previous step we've created a pool, but we haven't started it yet. So we do it in this step. + +1. Create a pipeline + +```bash +pachctl create pipeline -f toloka_wait_pool.json +``` + +2. Wait for the pool to be annotated by Tolokers + +```bash +pachctl list job -p toloka_wait_pool +``` + + +## Assignments aggregation + +In the pool we set each heading to have been annotated by 5 different Tolokers. Now we need to aggregate them to have +one and only one category for each heading. We will use `crowd-kit` library: + +```bash +pachctl create pipeline -f toloka_aggregate_assignments.json +pachctl list job -p toloka_aggregate_assignments +``` + +## Further data processing +### Datasets concatenation + +Suppose we want to add labelled data to train datasets. In this step we will +concatenate our train dataset and just labelled dataset from Toloka: + +```bash +pachctl create pipeline -f concatenate_datasets.json +pachctl list job -p concatenate_datasets +``` + +### Training and testing Random Forest classifier + +In most cases we need to annotate data for further model training. In this step we are going to create a pipeline that get train and test data from different sources, concatenate them if necessary and train Random Forest classifier with evaluating accuracy and F1 score. + +```bash +pachctl create pipeline -f train_test_model.json +pachctl list job -p train_test_model +pachctl get file train_test_model@master:random_forest_test.json 1> random_forest_test.json +cat random_forest_test.json +``` diff --git a/example/concatenate_datasets.json b/example/concatenate_datasets.json new file mode 100644 index 0000000..1fa1eab --- /dev/null +++ b/example/concatenate_datasets.json @@ -0,0 +1,43 @@ +{ + "pipeline": { + "name": "concatenate_datasets" + }, + "description": "A pipeline that concatenates two datasets into one.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/concatenate_datasets.py", + "--datasets", "/pfs/clickbait_data/train.csv", "/pfs/toloka_aggregate_assignments/results.csv", + "--output", "/pfs/out/enriched_train.csv" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "join": [ + { + "pfs": { + "repo": "clickbait_data", + "glob": "/", + "join_on": "$1" + } + }, + { + "pfs": { + "repo": "toloka_aggregate_assignments", + "glob": "/", + "join_on": "$1" + } + } + ] + } +} diff --git a/example/configs/pool.json b/example/configs/pool.json new file mode 100644 index 0000000..088fac3 --- /dev/null +++ b/example/configs/pool.json @@ -0,0 +1,108 @@ +{ + "assignment_max_duration_seconds": 120, + "defaults": { + "default_overlap_for_new_tasks": 5, + "default_overlap_for_new_task_suites": 5 + }, + "filter": { + "and": [ + { + "category": "profile", + "key": "languages", + "operator": "IN", + "value": "EN" + }, + { + "or": [ + { + "category": "computed", + "key": "client_type", + "operator": "EQ", + "value": "TOLOKA_APP" + }, + { + "category": "computed", + "key": "client_type", + "operator": "EQ", + "value": "BROWSER" + } + ] + } + ] + }, + "may_contain_adult_content": false, + "mixer_config": { + "golden_tasks_count": 1, + "real_tasks_count": 4, + "training_tasks_count": 0 + }, + "private_name": "Is this headline clickbait?", + "quality_control": { + "configs": [ + { + "collector_config": { + "parameters": { + "fast_submit_threshold_seconds": 30, + "history_size": 5 + }, + "type": "ASSIGNMENT_SUBMIT_TIME" + }, + "rules": [ + { + "action": { + "parameters": { + "duration_unit": "PERMANENT", + "private_comment": "bad quality", + "scope": "POOL" + }, + "type": "RESTRICTION_V2" + }, + "conditions": [ + { + "key": "fast_submitted_count", + "operator": "GTE", + "value": 2 + } + ] + } + ] + }, + { + "collector_config": { + "parameters": { + "history_size": 10 + }, + "type": "GOLDEN_SET" + }, + "rules": [ + { + "action": { + "parameters": { + "duration_unit": "PERMANENT", + "private_comment": "bad quality", + "scope": "POOL" + }, + "type": "RESTRICTION_V2" + }, + "conditions": [ + { + "key": "golden_set_correct_answers_rate", + "operator": "LTE", + "value": 90.0 + }, + { + "key": "golden_set_answers_count", + "operator": "GTE", + "value": 1 + } + ] + } + ] + } + ], + "training_requirement": { + "training_passing_skill_value": 90 + } + }, + "reward_per_assignment": 0.05 +} diff --git a/example/configs/project.json b/example/configs/project.json new file mode 100644 index 0000000..0c8b80e --- /dev/null +++ b/example/configs/project.json @@ -0,0 +1,37 @@ +{ + "assignments_issuing_type": "AUTOMATED", + "public_description": "Look at the a news headline and decide if it is clickbait or not.", + "public_instructions": "

About the task

\nIn this task you need to classify headlines on 2 categories: Clickbait or Not clickbait.
\n

What is Clickbait headline?

\nClickbait refers to the practice of writing sensationalized or misleading headlines.\nClickbait headline designed to make readers want to click on a hyperlink\nespecially when the link leads to content of dubious value. Typically clickbait titles cover not very useful content,\nso visitors tend not to stay for too long, that's why it's bad.\n", + "public_name": "Is this headline clickbait?", + "task_spec": { + "input_spec": { + "headline": { + "hidden": false, + "required": true, + "type": "string" + } + }, + "output_spec": { + "category": { + "hidden": false, + "required": true, + "type": "string" + } + }, + "view_spec": { + "config": "{\n \"view\": {\n \"items\": [\n {\n \"content\": {\n \"items\": [\n \"Headline: \",\n {\n \"path\": \"headline\",\n \"type\": \"data.input\"\n }\n ],\n \"type\": \"helper.join\"\n },\n \"type\": \"view.text\"\n },\n {\n \"data\": {\n \"path\": \"category\",\n \"type\": \"data.output\"\n },\n \"options\": [\n {\n \"value\": \"clickbait\",\n \"label\": \"Clickbait\"\n },\n {\n \"value\": \"notclickbait\",\n \"label\": \"Not clickbait\"\n }\n ],\n \"validation\": {\n \"hint\": \"you need to select one answer\",\n \"type\": \"condition.required\"\n },\n \"type\": \"field.button-radio-group\"\n }\n ],\n \"type\": \"view.list\"\n },\n \"plugins\": [\n {\n \"layout\": {\n \"kind\": \"scroll\",\n \"taskWidth\": 300\n },\n \"type\": \"plugin.toloka\"\n },\n {\n \"1\": {\n \"data\": {\n \"path\": \"category\",\n \"type\": \"data.output\"\n },\n \"payload\": \"clickbait\",\n \"type\": \"action.set\"\n },\n \"2\": {\n \"data\": {\n \"path\": \"category\",\n \"type\": \"data.output\"\n },\n \"payload\": \"notclickbait\",\n \"type\": \"action.set\"\n },\n \"type\": \"plugin.hotkeys\"\n }\n ]\n}", + "lock": { + "action.set": "1.0.0", + "condition.required": "1.0.0", + "core": "1.0.0", + "field.button-radio-group": "1.0.0", + "helper.join": "1.0.0", + "plugin.hotkeys": "1.0.0", + "plugin.toloka": "1.0.0", + "view.list": "1.0.0", + "view.text": "1.0.0" + }, + "type": "tb" + } + } +} diff --git a/example/configs/training.json b/example/configs/training.json new file mode 100644 index 0000000..a0f4e10 --- /dev/null +++ b/example/configs/training.json @@ -0,0 +1,11 @@ +{ + "private_name": "clickbait training", + "may_contain_adult_content": false, + "assignment_max_duration_seconds": 1800, + "mix_tasks_in_creation_order": false, + "shuffle_tasks_in_task_suite": false, + "training_tasks_in_task_suite_count": 10, + "task_suites_required_to_pass": 10, + "retry_training_after_days": 10, + "inherited_instructions": true +} diff --git a/example/create_toloka_pool.sh b/example/create_toloka_pool.sh new file mode 100755 index 0000000..fa4266b --- /dev/null +++ b/example/create_toloka_pool.sh @@ -0,0 +1,8 @@ +pachctl create repo toloka_pool +pachctl put file toloka_pool@master:pool.json -f ./configs/pool.json +pachctl put file toloka_pool@master:control_tasks.csv -f ./data/control_tasks.csv +pachctl put file toloka_pool@master:pool_tasks.csv -f ./data/pool_tasks.csv +pachctl list file toloka_pool@master +pachctl create pipeline -f toloka_create_pool.json +sleep 15s +pachctl list job -p toloka_create_pool diff --git a/example/create_toloka_pool_tasks.sh b/example/create_toloka_pool_tasks.sh new file mode 100755 index 0000000..c7cabc4 --- /dev/null +++ b/example/create_toloka_pool_tasks.sh @@ -0,0 +1,7 @@ +pachctl create pipeline -f toloka_create_control_tasks.json +sleep 15s +pachctl list job -p toloka_create_control_tasks + +pachctl create pipeline -f toloka_create_pool_tasks.json +sleep 15s +pachctl list job -p toloka_create_pool_tasks diff --git a/example/create_toloka_project.sh b/example/create_toloka_project.sh new file mode 100755 index 0000000..9039765 --- /dev/null +++ b/example/create_toloka_project.sh @@ -0,0 +1,6 @@ +pachctl create repo toloka_project_config +pachctl put file toloka_project_config@master:project.json -f ./configs/project.json +pachctl list file toloka_project_config@master +pachctl create pipeline -f toloka_create_project.json +sleep 15s +pachctl list job -p toloka_create_project diff --git a/example/create_toloka_training.sh b/example/create_toloka_training.sh new file mode 100755 index 0000000..d38498b --- /dev/null +++ b/example/create_toloka_training.sh @@ -0,0 +1,12 @@ +pachctl create repo toloka_training_config +pachctl put file toloka_training_config@master:training.json -f ./configs/training.json +pachctl list file toloka_training_config@master +pachctl create pipeline -f toloka_create_training.json +sleep 15s +pachctl list job -p toloka_create_training + +pachctl create repo toloka_training_tasks +pachctl put file toloka_training_tasks@master:training_tasks.csv -f ./data/training_tasks.csv +pachctl create pipeline -f toloka_create_training_tasks.json +sleep 15s +pachctl list job -p toloka_create_training_tasks diff --git a/example/data/control_tasks.csv b/example/data/control_tasks.csv new file mode 100644 index 0000000..6c4113e --- /dev/null +++ b/example/data/control_tasks.csv @@ -0,0 +1,21 @@ +INPUT:headline,GOLDEN:category +Barack Obama accepts US presidential nomination from the Democratic Party,notclickbait +Senator Says He Had Affair With an Aide,notclickbait +Controversial cancer test gains support,notclickbait +Clashes in France after anti-condom speech by Pope,notclickbait +Answer These Four Questions And We'll Guess Your Gender,clickbait +25 Feminist Songs That Always Make You Feel Like A Boss,clickbait +Indiana businessman held hostage in Iraq,notclickbait +Santas Guess Bizarre Candy Cane Flavors,clickbait +Russia's Nomadic Reindeer Herders Face The Future,clickbait +Eurozone now officially in recession,notclickbait +Pacific Rim braces for tsunami following major Chilean earthquake,notclickbait +"Phoebe From ""Friends"" Is Literally The Fucking Worst",clickbait +Geffen Said to Be Still Interested in The New York Times,notclickbait +RTÉ's Eddie Hobbs attracts massive audience,notclickbait +It's Time To Talk About How The Recast Dumbledore Was Literally The Worst,clickbait +19 Things Anyone Who's Best Friends With Their Mum As An Adult Will Understand,clickbait +We Know Your Favorite Music Genre Based On Your Zodiac Sign,clickbait +"Dannie Abse hurt, wife killed in car accident",notclickbait +Football Vs. Wife: Which Do You Know Better,clickbait +Everything You Need To Know About This Chocolate Cheese Toastie,clickbait diff --git a/example/data/pool_tasks.csv b/example/data/pool_tasks.csv new file mode 100644 index 0000000..2a776ec --- /dev/null +++ b/example/data/pool_tasks.csv @@ -0,0 +1,31 @@ +INPUT:headline +Tamil party drops commitment to independence from Sri Lanka +"The Hardest ""Harry Potter"" Spelling Test You'll Ever Take" +Two women first ever to serve on municipal council in Kuwait +Cattle Rustling Plagues Ranchers +Are You More Cara Delevingne Or Taylor Swift +"Jennifer Lawrence Wore A Jacket That Says ""Perv""" +Ten climbers killed in avalanche at Turkish ski resort +"How To Draw 4 Badass ""Star Wars"" Helmets" +Protesters against police violence surround London's Scotland Yard +"15 ""Gossip Girl"" Book Plotlines That Never Made It To The Show" +Chip Ganassi makes American motor sports history +This Lingerie Company's New Model Is Unapologetically Curvy +No Simple Route for Flight Delay Information +Australia celebrates Australia Day 2009 +People Are Mourning David Bowie On Twitter And It's Both Moving And Heartbreaking +Arkansas Democratic party chairman assassinated by gunman +Are You More Halsey Or Melanie Martinez +Damn Daniel Has Turned A Teenager Into A Huge Meme +Norway Slides Into Recession +Odd Paths Lead the Way to N.B.A. Riches +"Accounts, People & Miscellany" +"A Guy Just Did The Most Epic ""Cha Cha Slide"" Dance Ever" +Mo Ibrahim: Some African countries too small to continue to exist independently +Andrew Sayers resigns National Museum of Australia directorship +For Everyone Who Wasn't Allowed To Celebrate Halloween +We Know Whether You'll Be Alone On Valentine's Day +Test Your Random Knowledge With This Themeless Crossword +Are You More Queen Elsa Or Jack Frost +Indian Wells Officials Having No Luck With Williams Sisters +21 Signs You're In A Relationship With Your Dog diff --git a/example/data/test.csv b/example/data/test.csv new file mode 100644 index 0000000..57fb2c7 --- /dev/null +++ b/example/data/test.csv @@ -0,0 +1,601 @@ +headline,category +Obama Promises the World a Renewed America,notclickbait +"Would You Receive The Final Rose From ""Bachelor"" Ben Higgins",clickbait +Argentine President harshly criticizes US border fence,notclickbait +Terror suspects arrested in connection with bombing public bus in Israel,notclickbait +Obama Pledges Attention to Flooding,notclickbait +Zenit rocket launches Galaxy 18 satellite,notclickbait +What Are Your Best Tips For Keeping The Weight Off,clickbait +This Guy Celebrated A Bachelor Party Alone After The Rest Of The Wedding Party Got Stranded,clickbait +21 Stores You'll Love If You Are Addicted To Urban Outfitters,clickbait +Chinese Golfer Makes a Name for Herself on L.P.G.A. Tour,notclickbait +"John Kerry criticizes news media, joins call for inquiry on Downing Street Memo",notclickbait +Obama Telephones Afghan President,notclickbait +19 Small Awards Anyone With Anxiety Deserves To Receive,clickbait +"Meet Dhee, Bangladesh's First Lesbian Comic Book Character",clickbait +"Florida Speaker Steps Down, Citing Corruption Investigation",notclickbait +We Must Never Forget That Ben Affleck And Hillary Swank Were In The Best Movie Ever,clickbait +Miles Franklin Literary Award nominates only women for potential winners,notclickbait +Somali man attempts failed attack on controversial cartoonist,notclickbait +The Drake Relays: a Big-Time Meet in a Small-Town Atmosphere,notclickbait +"Stricker Leaves Opening, and Clark Slips Into Lead at Colonial",notclickbait +Republican Congressman Ron Paul endorses Constitution Party nominee Chuck Baldwin for President of the United States,notclickbait +Qatar Said to Be Pondering a Stake in Porsche,notclickbait +New Mexico Legislature Passes Caps on Contributions,notclickbait +"Let's See How Well You Remember The First Episode Of ""Misfits""",clickbait +Boeing 767 cargo plane seriously damaged by fire at San Francisco,notclickbait +30 Decadent Fall Desserts For People Who Don't Like Pumpkin,clickbait +British military secrets leaked on social networking sites,notclickbait +14 Life-Changing Ways To Bake With Avocados,clickbait +Family and Lawyer Fear for Reporter Who Threw His Shoes at Bush,notclickbait +German football: Lahm's contract offer withdrawn; Schlaudraff to Hannover,notclickbait +McNamee Says He Injected Clemens at Yankee Stadium,notclickbait +Last-Second Shot Sends Villanova Past Pitt and Into Final Four,notclickbait +"In Recession, Strategy Shifts for Big Chains",notclickbait +When You Think You Might Snap,clickbait +"In Gaza, Weighing Crimes and Ethics in Urban Warfare",notclickbait +Wikimania 2007 Site Events warming up for main conference,notclickbait +This Heart Test Will Determine What Kind Of Romantic You Are,clickbait +Media Maelstrom Awaits Alex Rodriguez in Florida,notclickbait +Man buys his stolen camera on eBay,notclickbait +"Phoebe From ""Friends"" Is Literally The Fucking Worst",clickbait +BHP Closing Nickel Mine as Commodity Prices Fall,notclickbait +Environmentalists in a Clash of Goals,notclickbait +Iran announces its building of a second nuclear power plant,notclickbait +New host announced for 2007 Netball World Championships,notclickbait +Can You Identify Which Disney Channel Movie Each Celebrity Was In,clickbait +Non-Mexicans Try To Guess The Meaning Of Mexican Slang,clickbait +Guatemalan president and first lady granted divorce,notclickbait +Remember When Anderson Cooper Was A Reality TV Host,clickbait +Jailed former Taiwanese President on hunger strike,notclickbait +Did You Peak In 2015,clickbait +Do You Belong In London Or San Francisco,clickbait +Could You Actually Quit Facebook,clickbait +NATO to expand Afghanistan presence,notclickbait +What's The Biggest Misconception About Being A Military Spouse,clickbait +"As Protesters Pause in Thailand, Their Grievances Against Elite Simmer",notclickbait +Isle of Man Plans Unlimited Music Downloads,notclickbait +Putting U.S. Trucking on a Diet,notclickbait +18 Ways To Avoid The Holiday Blues This Year,clickbait +31 Hilarious Times Tumblr Explained Our Weird Brains,clickbait +Queen RiRi Might Be Dating Travis Scott,clickbait +"Even at Carolina, Penguins Dominate Hurricanes",notclickbait +Iraqi provincial elections relatively peaceful,notclickbait +People Try Bizarre Candy Canes,clickbait +First Prime Minister of Greenland Jonathan Motzfeldt dies at age 72,notclickbait +29 Hilarious Fake Literary Band Names,clickbait +This Phone Accessory Will Make Your Photos Look Sick As Hell,clickbait +U.S. Issues Scathing Report on Immigrant Who Died in Detention,notclickbait +Everyone's Favourite Childhood Anthem Just Got An Epic Makeover,clickbait +Are You More Donald Trump Or Donald Duck,clickbait +21 Tweets That Are Way Too Funny For Anyone With A Pregnant Sister,clickbait +18 Snapchats That Hit Too Close To Home,clickbait +ECOWAS suspends Niger in dispute over constitution,notclickbait +42 Very Urgent Questions I Have For Cats,clickbait +Taliban Accepts Pakistan Cease-Fire,notclickbait +Microsoft extends warranty for all Xbox 360s,notclickbait +18 Headaches Everyone Who Works In An Office Knows,clickbait +Lockerbie convict's family among protesters for justice in Edinburgh,notclickbait +If Siri Were A Real Personal Assistant,clickbait +You've Been Making Pancakes Wrong Your Entire Life,clickbait +"17 ""Princess Bride"" Gifts You Need In Your Life",clickbait +2-year-old dies in car in 100ºF heat with windows rolled up,notclickbait +"Harlan Ellison sues CBS-Paramount, WGA over Star Trek royalties",notclickbait +"This Cover Of AC/DC's ""Thunderstruck"" On Bagpipes Is Fierce AF",clickbait +23 Of The Most Satisfying Gaming Achievements,clickbait +Off to Practice Before a Round of Golf? Think Again,notclickbait +"Poll: Arabs discouraged by US policies, back nuclear-armed Iran",notclickbait +16 Photos That Capture What It's Like Renting In Toronto,clickbait +"Kristen Stewart Talks About Honesty, Privacy, And Her New Movie ""Equals""",clickbait +"Eritrea plans to vaccinate 500,000 children",notclickbait +Greece formally asks for EU-IMF loans,notclickbait +Australian and New Zealander freemasons arrested for sorcery in Fiji,notclickbait +Iraqi Policemen to Face Charges of Prison Abuse,notclickbait +An Industry That Sells Takes a Look in the Mirror,notclickbait +Can You Guess Which Celeb Made This Art,clickbait +Guys Transform Into Pinups,clickbait +17 Desserts Guaranteed To Turn You Into A Chocoholic,clickbait +How Much Do You Really Love Your Dog,clickbait +19 Sibling Moments That Are Too Real For Latinos,clickbait +Denver win 2009 USAFL National Championship,notclickbait +19 Gifts Every Classical Music Nerd Will Love,clickbait +"Should You Marry Lucas Or Nathan Scott From ""One Tree Hill""",clickbait +This Mattress Has A Built-In Pet Bed Because The Snuggle Is Real,clickbait +Watch This Woman Style A Hijab In 7 Easy Ways,clickbait +Iraq Leader Omits a Bit in Lauding U.S. Pullout,notclickbait +Hovind's 11th Circuit Court Appeal Denied,notclickbait +Harry Potter Is Real: Researcher Uncovers Real-Life Professor Snape At Welsh University,clickbait +Literally Just A Bunch Of Inspiring GIFs Of The Rock,clickbait +Can You Handle Alcoholic Oreos,clickbait +"This Is What It's Like To Eat Durian, The Stinky And Scary King Of Fruits",clickbait +Everything You Need To Know About The Sweetest Bakery In London,clickbait +How Much Do You REALLY Know About Cake,clickbait +How Lazy Are You Being Today,clickbait +We Know Your Best Quality Based On Random Questions,clickbait +"Tropical Storm Adrian forms in Eastern Pacific, threatens Central America",notclickbait +17 Images You Won't Be Able To Unsee,clickbait +Elizabeth II annuls Fred Goodwin knighthood,notclickbait +Can You Name These Muppets,clickbait +17 Pictures Hot People Will Never Understand,clickbait +New prince is born in Norway,notclickbait +William Morris and Endeavor Explore a Merger,notclickbait +"Five children found dead in Graham, Washington",notclickbait +How To Cook A Real Feast Of The Seven Fishes,clickbait +15 Hits From 1996 That'll Make You Feel Old AF,clickbait +Manchester City agree big money for Shaun Wright-Phillips to move to Chelsea FC,notclickbait +Former Taiwanese president jailed for life on corruption charges,notclickbait +Take A Look Back At The Biggest Black Friday Toys Of The Past 20 Years,clickbait +Dima Bilan wins the 2008 Eurovision Song Contest for Russia,notclickbait +Science of champagne bubbles explained,notclickbait +23 Times Tumblr Cleverly Explained What Being Genderqueer Means,clickbait +"Should RealNetworks, an Internet Pioneer, Switch Gears?",notclickbait +"In Horse Racing, Pedigree Is Not a Guarantee of Breeding",notclickbait +19 Insane Milkshake Shops Around The World You Need To Eat At,clickbait +"How Well Do You Remember ""30 Rock""",clickbait +Cool front to bring lasting relief for Texas,notclickbait +"27 Times Tiffany From ""Daria"" Was Iconic AF",clickbait +HP France in trouble due to alleged monopolist behavior,notclickbait +People Who Don't Swear,clickbait +"Man bites dog, charged with animal cruelty",notclickbait +Pinterest Is Actually A Porn Director's Best Friend,clickbait +"U.S. Looks at Dropping a Condition for Iran Nuclear Talks, Officials Say",notclickbait +Jenson Button wins 2009 Bahrain Grand Prix,notclickbait +19 Things That Prove Being Beautiful Can Be Scary,clickbait +Guinea to launch investigation into killing of protestors,notclickbait +"Which ""Percy Jackson"" Demigod Are You",clickbait +Mark Fidrych Remembered for Remarkable Season and Endearing Antics,notclickbait +Orrin Hatch accidentally compares Iraq to Vietnam,notclickbait +Are You More Ja Rule And Ashanti Or Ja Rule And J.Lo,clickbait +"UK group Liberty, Edinburgh city council on Scientology 'cult' signs",notclickbait +"The ABCs According To ""Friends""",clickbait +False cancer cure claims lead to federal charges against five US companies,notclickbait +19 Things That Happen When Your Song Comes On In The Club,clickbait +Everything You Need To Know About This Chocolate Cheese Toastie,clickbait +"Xenophobia Threatens Italy, President Giorgio Napolitano Warns",notclickbait +I Had My Hair And Makeup Done Like Kim Kardashian For A Week And Here's What Happened,clickbait +Uniting Church at Auburn destroyed by fire,notclickbait +"The Lake Wobegon Effect, in Actively Managed Funds",notclickbait +"London Tube bombs went 'bang bang bang, very close together'",notclickbait +Exeem Annonunced To Be Successor Of Suprnova,notclickbait +Who Said It: Chandler Bing Or Liz Lemon,clickbait +Diane Keaton Running To Get Froyo Is Diane Keaton Living Her Best Life,clickbait +BuzzFeed Writers Try To Do Handstands,clickbait +28 Bros Who Absolutely Crushed 2015,clickbait +Philippines welcomes world's 7 billionth baby,notclickbait +"11 Questions Every ""Doctor Who"" Fan Has, Answered By Steven Moffat",clickbait +"The New Trailer For ""The Secret Life Of Pets"" Is Here And It's Adorable",clickbait +"Jessie J Sang Little Mermaid's ""Part Of Your World"" And It's Insane",clickbait +'Mobile phone dermatitis' linked to nickel deposits,notclickbait +Deadly virus samples missing in Mexico/Lebanon,notclickbait +13 Pets Who Don't Understand How Cute They Are,clickbait +"NTSB continues investigation of near-collision in Pennsylvania, United States",notclickbait +15 Words That Have A Totally Different Meaning To Your Mom,clickbait +US combat forces pull out of Iraq,notclickbait +"These 24 Moments From ""Pride & Prejudice"" Will Make Your Heart Melt",clickbait +Singles Try To Explain Cuffing Season,clickbait +Bush nominates John G. Roberts for U.S. Supreme Court,notclickbait +The One Way To Get A Phone Number,clickbait +"These Pictures From The ""Fuller House"" Set Will Make You Extremely Nostalgic",clickbait +"15 Times ""She's The Man"" Accurately Portrayed High School Life",clickbait +"Jack Herrick, wikiHow founder interviewed by Wikinews",notclickbait +Unlikely Ally for Residents of West Bank,notclickbait +A Bunch Of Women Wore Pants To The Emmys And It Was Perfect,clickbait +Can You Find All 23 Chameleons Hiding In This Forest,clickbait +What Gift Should You Buy Your Favorite Geek,clickbait +High tides sweep west Galveston Bay,notclickbait +Can You Guess What Happens When You Try To Eat 50 McDonald's Nuggets In 5 Minutes,clickbait +Gunman on the loose in Melbourne,notclickbait +"Baby survives after stroller hit by train in Melbourne, Australia",notclickbait +18 Stunning Christmas Cookies Guaranteed To Impress Your Family,clickbait +Australian politician Karen Overington dies aged 59,notclickbait +Israeli shelling on UN school on the Gaza Strip kills six,notclickbait +Spanish newspaper co-founder Carlos Mendo dies aged 77,notclickbait +UK loses appeal to conceal Binyam Mohamed torture,notclickbait +18 Celebrity Snapchat Accounts That'll Change Your Snap Life,clickbait +"41,000 Indians Have Petitioned The Censor Board For More Gender-Sensitive Ratings",clickbait +"2,000 face redundancy at English steelworks",notclickbait +"After a Week of Travel, Castroneves Has a New Trip",notclickbait +Can You Identify The Valentine's Chocolates (Without Tasting Them),clickbait +Coldplay's new album hits stores worldwide this week,notclickbait +Can You Spell The Name Of This Welsh Village,clickbait +Canadian military accused of Agent Orange cover up,notclickbait +Former First Lady of Taiwan Admits Laundering $2.2 Million,notclickbait +"PSA: There's A Chrome Extension That Blocks ""Star Wars"" Spoilers",clickbait +"Adam Woodyatt Is Upset A #HowsAdam Moment Didn't Happen During The ""Corrie"" Live Episode",clickbait +I Was Added To A Group Chat With Eight Strangers And It Got Incredibly Weird,clickbait +Can You Guess The Lana Del Rey Music Video From The Screenshot,clickbait +Tyra Banks Announced The Birth Of Her First Child In The Most Adorable Way,clickbait +"The Dark Knight film director's brother arrested for murder, kidnapping",notclickbait +"23 Times Andy Dwyer From ""Parks & Rec"" Was Wrong About Everything",clickbait +"UN: Ethiopian GDP grew only 1.7% in 2009, may not reach anti-poverty goals",notclickbait +17 Times The Internet Nailed What It's Like To Be A Pisces,clickbait +These Mini Cheesecakes Will Make You So Content With Life,clickbait +Row over Lech Wałęsa's alleged collaboration with communists escalates,notclickbait +23 Weird As Hell U.S. Laws You Won't Believe Are Real,clickbait +Why You Should Be Proud To Be A Slytherin,clickbait +Trustee in Madoff Case Processing 100 Cases a Week,notclickbait +17 Ways To Procrastinate At Work Without Getting Caught,clickbait +West Virginia Governor Is Urged to Add Financing for Mental Health Care,notclickbait +Russian polar submarine TV footage faked,notclickbait +Forget the Cigarette Lighter Adapter. Now You Can Dock Your iPod in the Car Radio.,notclickbait +17 Ways To Keep Your Kid Safe This Halloween,clickbait +19 Struggles Of Taking Your Kid To The Bathroom In Public,clickbait +Republicans Firing Blanks at Obama,notclickbait +Suicide bomber kills five Afghan children,notclickbait +Things Nobody Told Me About Depression,clickbait +"Which ""AHS"" Character Are You Based On Your Choice Of Cake",clickbait +Alabama School Bus Crash kills 4,notclickbait +"Very Lonely Luke Is The Latest Hilarious ""Star Wars"" Parody Twitter Account",clickbait +31 Tweets About Fast Food That Will Actually Make You Laugh,clickbait +"Rail explosion reported at Viareggio, Italy",notclickbait +Eleven die in truck-van crash in Kentucky,notclickbait +Adele Is Currently Experiencing What She Put All Of Us Through For The Last Four Years,clickbait +13 Things To Do For Your Friend With Post-Natal Depression,clickbait +Warner Brothers Plans to Cut 800 Jobs,notclickbait +14 Celebrity Names You ALWAYS Have To Google,clickbait +Egyptian president will not seek re-election in September after protests,notclickbait +Can We Guess Your Personality Based On Your Froyo Order,clickbait +"Everyone's Freaking Out Over The Season Premiere Of ""Arrow""",clickbait +"17 Reasons Nadiya From ""Bake Off"" Was The Best British Person 2015",clickbait +Profit Dropped 14% in Quarter for Staples,notclickbait +Industry Ignored Its Scientists on Climate,notclickbait +Air France jet with 228 on board goes missing,notclickbait +Rainy Days in Paradise,notclickbait +Final draw sets groups for FIFA World Cup 2010,notclickbait +"Madoffs Shared Much, but How Much?",notclickbait +2007 German League Cup: Bayern Munich wins 6th League Cup,notclickbait +Potterheads Try Fan Created Sweets,clickbait +Turkey mourns flotilla dead,notclickbait +What It's Like To Write The Most Hyped Book Of The Year,clickbait +"North, South Korea plan united team for 2008 Olympics",notclickbait +"Football Aside, Tom Brady Takes Some Really Cute Dad Pics",clickbait +At least 85 dead in shooting at Norwegian youth camp,notclickbait +An Adorable Animals Advent Calendar: December 8,clickbait +This Is What Disney Princesses Would Look Like In Real Life,clickbait +19 Incredibly Important Reasons You Should Watch The Rugby World Cup,clickbait +"Your Self Care Routine Should Include BuzzFeed's ""Another Round"" Podcast",clickbait +18 Genius Ways To Eat Cheese For Every Meal,clickbait +"At Paris Air Show, Little Flash and Lots of Introspection",notclickbait +"Turns Out No One Knows The Words To ""Waterfalls""",clickbait +"8 dead following road collision in New Brunswick, Canada",notclickbait +Papua New Guinea Culture and Tourism minister charged with attempted murder,notclickbait +Deaths on South African roads at 1215 for December 2005,notclickbait +Many Gulls Die After Oil Spills Into River in Ohio,notclickbait +17 Beavers Who Just Want To Be Your Feminist Ally,clickbait +"The One Person You Probably Never Noticed In Britney Spears' ""Piece Of Me"" Music Video",clickbait +Which Cool Outer Space Shit Are You Based On Your Zodiac Sign,clickbait +Deal Is Reached to Keep Boston Sailing Show Afloat,notclickbait +Oestgaard Buaas Retains U.S. Slopestyle Title,notclickbait +"Man dies after being hit by train in Moray, Scotland",notclickbait +"29 Of The Most Perfect Comebacks That Ever Happened On ""Friends""",clickbait +"How Well Do You Remember ""Freaky Friday""",clickbait +Russia raises minimum vodka prices,notclickbait +Which Of Justin Bieber's Hairstyles Are You,clickbait +15 Pieces Of Sushi That Are A Sin Against Nature,clickbait +Salma Hayek Had To Go To The ER In An Embarrassing T-Shirt,clickbait +Neville Chamberlain's War Diaries go on display,notclickbait +25 More Tweets About Animals That Will Make You Laugh Every Time,clickbait +Tropical Storm Ignacio forms over the Pacific,notclickbait +25 Of 2015's Highest-Paid Musicians According To Forbes,clickbait +Kathy Griffin And Kristin Chenoweth Talk About Their Worst Heckler Stories,clickbait +Israeli army reoccupies Tulkarem in West Bank,notclickbait +"Africa makes first draft version of UNCCC treaty, with harder goals",notclickbait +"Obama's first State of the Union speech focuses on economy, jobs",notclickbait +"If Social Media Ever Gets You Down, This Pop Song Is For You",clickbait +Here's How To Get A Haircut You Actually Like,clickbait +Can You Pass This Texas Food Test,clickbait +19 Times Canadians Were Condescending To Americans About Winter,clickbait +19 Signs You're Finally Becoming An Adult,clickbait +"Wikinews interviews Brian Moore, Socialist Party USA presidential candidate",notclickbait +7 Reasons Rey Is Definitely Luke Skywalker's Daughter,clickbait +Aziz Ansari's New Show Has Its First Trailer And It's Pretty Damn Great,clickbait +8 Ways To Style Your Hair For The Gym That Are Actually Awesome,clickbait +Tools That Leave Wildlife Unbothered Widen Research Horizons,notclickbait +"Calls for aid to help feed millions, as East Africa plunges into drought",notclickbait +Indiana businessman held hostage in Iraq,notclickbait +What's The Strangest Way You've Lost A Tooth,clickbait +17 Practical Items That Will Soothe Your Type-A Soul,clickbait +34 Eye Opening Photos Of The Great Depression,clickbait +2007 FIFA U-17 World Cup: Germany tops Group F,notclickbait +"Watch What Happens When Teens React To ""Back To The Future Part II""",clickbait +"23 Hilarious Hidden Jokes On ""The Simpsons"" You Probably Missed",clickbait +Ellen DeGeneres As A Member Of One Direction Is Too Perfect,clickbait +15 Things You Shouldn't Say To Bostonians,clickbait +Can You Pass This Sunday School Test For Children,clickbait +26 Beauty Products Our Readers Loved In 2015,clickbait +H.P. Lowers Bar for Printing Glossy Color Magazines,notclickbait +"This Is The Hardest ""Divergent"" Quiz You'll Ever Take",clickbait +Couples Who Prove Opposites Attract,clickbait +This Optimistic Man Tried To Sneak 14 Bottles Of Liquor Into Saudi Arabia In His Underwear,clickbait +Which Dog Breed Are You Based On Your Zodiac Sign,clickbait +"Big Brown victorious in Kentucky Derby, runner-up Eight Belles breaks down",notclickbait +This Dude Quit His Job To Train Full-Time To Break The World Record For Most Selfies In An Hour,clickbait +Rebels take over South Sudan oil regions,notclickbait +Iran close to decision on nuclear program,notclickbait +"The First ""Hunger Games"" Red Carpet Vs. The Last",clickbait +15 Dogs Who Are Having Their Greatest Birthday Ever,clickbait +21 Kitchen Gadgets That Actually Help You Eat Healthier,clickbait +The 24 Funniest Tweets About Cats In 2015,clickbait +Two Young Actors Give The Year's Most Heartachingly Mature Performances,clickbait +16 Italian Landmarks That Are Actually Crap,clickbait +Fake impotence drugs linked to low blood sugar outbreak,notclickbait +Is This A Close Up Of Eggnog Or A Handful Of Mayonnaise,clickbait +Pirates seize cargo ship in Indian Ocean,notclickbait +Anonymous people reveal animal cruelty at Australian Inghams poultry producer factory using CCTV footage,notclickbait +American Samoa received eight minutes warning before 2009 tsunami,notclickbait +San Francisco hit-and-run suspect caught after lying,notclickbait +This 8-Picture Test Will Reveal Your Greatest Quality,clickbait +A Defeat That Fits the Knicks to a T,notclickbait +Canadian PM will shuffle cabinet today,notclickbait +Is Your Force Awakened,clickbait +Balancing Freedom and the Role of the State in Germany,notclickbait +British Regulator Accuses 3 of Insider Trading,notclickbait +"Which Hogwarts Houses Do The ""PLL"" Characters Belong In",clickbait +Can We Guess If You Are Team Peeta Or Team Gale Based On Your Zodiac Sign,clickbait +8 Signs You Were Raised In The South,clickbait +Clinton Says U.S. Is Ready to Lead on Climate,notclickbait +U.S. Seeks to Reduce Ship Emissions in Coastal Areas,notclickbait +11 Reasons Leslie Knope Is The Queen Of Compliments,clickbait +"Nortel to Cut Another 3,200 Jobs",notclickbait +"Swimmer Michael Phelps of the U.S. wins first gold medal of 2008 Summer Olympics, breaks world record",notclickbait +"Try To Stay Calm '90s Kids, But The ""Full House"" House Is Being Reconstructed",clickbait +Which Pop Anthem Matches Your Birth Month,clickbait +"Are Justin Trudeau And ""American Horror Story"" Star Finn Wittrock The Same Person",clickbait +23 Pictures That Will Make All Responsible Adults Extremely Angry,clickbait +15 Rude Texts Your Tears Would Send You,clickbait +Founder of UK sports car manufacturer TVR dies in Spain,notclickbait +Jackie Chan Strikes a Chinese Nerve,notclickbait +This 11-Year-Old Girl Started A Project Called #1000BlackGirls To Get More Diverse Books In Schools,clickbait +11 Ridiculous Selfie Confessions,clickbait +Obama succeeds Bush as 44th president of the United States,notclickbait +Football Vs. Wife: Which Do You Know Better,clickbait +21 Minimalist Ways To Store Everything In Your Home,clickbait +23 Things That Could Only Happen In San Francisco,clickbait +Turkish writer Orhan Pamuk receives Nobel Prize,notclickbait +This Girl Used Makeup To Turn Herself Into Gigi Hadid And It's So Good,clickbait +17 Struggles Every Clean Roommate Knows To Be True,clickbait +"I Let Twitter Run My Life For A Day, And Here's What Happened",clickbait +Somali Group Said to Hold 2 Frenchmen,notclickbait +11 Startling Boob Confessions,clickbait +Pakistani Reporter Is Killed After Rally for Taliban Truce,notclickbait +Protesters Defy Iranian Efforts to Cloak Unrest,notclickbait +Proposal to Raise Income Tax in Pennsylvania,notclickbait +Have You Had A Real-Life Meet-Cute,clickbait +"There's A New Trailer For The ""Sherlock"" Christmas Special And It's Pure Magic",clickbait +"Here's Why ""Paw Patrol"" Is A Terrible Kids' Show",clickbait +"HiNet, WCG, and WGT unite holding Taiwanese Gaming Athletes' Qualification",notclickbait +Ben Shephard announces departure from GMTV,notclickbait +Tornado jet crashes in Scotland,notclickbait +Google now worth over $80 billion,notclickbait +UN emphasizes importance of women's health in Africa,notclickbait +"I'm Queer, The Mormon Church Doesn't Want Me, But I'm Staying",clickbait +Triathlon national championship held in Belgium,notclickbait +Talk-therapy can make a difference in early treatment of severe depression,notclickbait +Should You Buy Apple's New iPhone Battery Case,clickbait +Islamabad on red alert for possible terrorist attack,notclickbait +"Sure-Footed, Boonen Captures His 3rd Paris-Roubaix",notclickbait +Deadly flooding in Pakistan kills hundreds,notclickbait +Former Japanese prime minister Hashimoto indicated his retirement,notclickbait +Football: Fulham FC sacks coach Chris Coleman,notclickbait +Politicians call for NY Sen. Monserrate to resign after assault conviction,notclickbait +29 Celebrities In 2006 Compared With Now,clickbait +Teens sought in Pennsylvania killings found,notclickbait +Golf: Harrington wins British Open,notclickbait +Bomb in Dagestan explodes Russian military truck,notclickbait +Jennifer Lawrence Is More Excited About The New Kardashian Baby Than You Are,clickbait +North Korea seeks diplomatic relations with the US,notclickbait +We Surprised Our Coworkers With BB-8 And Here's What Happened,clickbait +Broadband users kicked off service for constant questioning,notclickbait +Surf's up in Chile; championship competition in Pichilemu,notclickbait +Dick Cheney makes surprise Iraq visit,notclickbait +Is This The Name Of A GOP Candidate Or An Ikea Couch,clickbait +Ancient prayer book found in Irish bog,notclickbait +U.S. Resumes Surveillance Flights Over Pakistan,notclickbait +"Rangers Shut Out Devils, Assisted by Avery",notclickbait +British student falls to his death from 10th floor window,notclickbait +24 Times Ruby Rose And Phoebe Dahl Defined Relationship Goals In 2015,clickbait +7 Healthy Eating Tricks You'll Actually Want To Try,clickbait +How Are You Feeling Right Now,clickbait +17 Relationship Goals That Are Actually Worth Achieving,clickbait +24 Spectacular One-Tier Wedding Cakes,clickbait +English Premier League: Week 33 round-up,notclickbait +"Its Population Falling, Russia Beckons Its Children Home",notclickbait +Pakistan to Turkey container train service launched,notclickbait +U.S. government proposes removing Yellowstone grizzlies from endangered species list,notclickbait +First Chinese tourists arrive in the UK,notclickbait +My Roommate Built A Cat Castle Fit For Royalty,clickbait +"13 ""Grease: Live"" Secrets I Learned While Sitting In The Audience",clickbait +A Rescue Dog And Her Handler Who Worked At Ground Zero During 9/11 Were Honored With An Amazing Day,clickbait +Extreme Phone Pinching Is Possibly The Most Nerve-Racking Internet Craze Yet,clickbait +Wikinews interviews author and filmmaker John Gaspard,notclickbait +Which Pie Matches Your Zodiac Sign,clickbait +Pirates capture Saudi oil tanker,notclickbait +"Diwali, As Explained By My Brown Dad",clickbait +Five Irish schoolgirls die in bus crash,notclickbait +UN official: 90 Rwandan rebels killed in DRC by army troops,notclickbait +"For Peugeot Chief, History Repeats Itself",notclickbait +NASA's Mars rovers exceed all expectations,notclickbait +29 Faces You'll Immediately Recognize If You've Ever Been To The Gyno,clickbait +"Reminder That Daniel Radcliffe's ""Harry Potter"" Audition Is Literally The Cutest Thing You'll Ever See",clickbait +"Dam in Queensland, Australia bursts, four missing",notclickbait +Irish Government sends team of officials in hunt for Journalist,notclickbait +It's Time For You To Make A Snap Decision: NYC Or LA,clickbait +Sprint/RealNetworks to provide cell phone Internet radio and podcasts in US,notclickbait +Landon Donovan the Striker Is Also a Lightning Rod,notclickbait +4 U.S. Soldiers Killed in Afghanistan,notclickbait +Can You Get Laid Tonight,clickbait +What Your Crush Says VS What You Think,clickbait +Can This Little Gadget Help Your Posture,clickbait +We Played With Bizarre Sex Toys So You Don't Have To,clickbait +"Orange County bus strike ends as union, board approve contract",notclickbait +Try And Guess What Halloween Candies These Were Before We Smashed Them,clickbait +"American swimmer Michael Phelps laments ""bad judgment"" in marijuana controversy",notclickbait +No people or animals hurt in rural Australian fire,notclickbait +Gay teachers' status uncertain after Polish election,notclickbait +I Returned To My Childhood Mall And Found A Nightmare,clickbait +French Minister Says Retirement Age Will Rise,notclickbait +"World's cheapest car launched in India, will go on sale in April",notclickbait +Which Taylor Swift Track Should Be Your Personal Theme Song,clickbait +Ex-Soldier Gets Life Sentence for Iraq Murders,notclickbait +Scientists develop 'Smart Bomb' treatment that targets cancer cells,notclickbait +House Bill Would Set Up Database for Artificial Joints,notclickbait +Scientists confirm new superheavy element,notclickbait +United States Senator Lieberman to speak at Republican party convention,notclickbait +"Continuing His Hot Start, Matt Kenseth Wins Auto Club 500",notclickbait +Want To Call It Frisco? Ask The Hells Angels,clickbait +Documents show U.S. knew of Guatemalan human rights abuses,notclickbait +Suicide bomber in southern Russia kills at least five policemen,notclickbait +Indonesian parliament approves privatising of three major state firms,notclickbait +US President Barack Obama speaks at memorial for Arizona shooting victims,notclickbait +17 Unusual And Beautiful Russian Baby Names,clickbait +Which Netflix Original Series Should You Star In,clickbait +"With Goals Tougher to Make, Rules Change on Bonuses",notclickbait +21 Times Red Forman Was The Realest Fucker On The Planet,clickbait +"EU bans all Indonesian airlines as well as several from Russia, Ukraine and Angola",notclickbait +23 Tweets That Prove You Should Never Cross Cara Delevingne,clickbait +Taiwan's cabinet resigns,notclickbait +Bus explosion in Pakistan kills at least 40,notclickbait +"Advice in Two New Books, but Not From Talking Heads",notclickbait +17 Uncommon Survival Products That Will Actually Change Your Life,clickbait +How Many Of These Books By YouTube Stars Have You Read,clickbait +18 Reasons We Don't Want Thanksgiving Break To Be Over,clickbait +"27 Funny, Bizarre, And Ingenious Excuses People Have Used To Skip School",clickbait +"F#@k, Marry, Kill: The ""Divergent"" Edition",clickbait +"Ryanair sack, sue pilot over participation in safety documentary",notclickbait +25 Reasons 2015 Was The Year Of The Geek,clickbait +Teaching Intelligent Design: Incumbent Dover PA school board fails reelection,notclickbait +Mob kills 'witches' in Kenya,notclickbait +Rare Middle East cyclone batters Oman,notclickbait +Marcinkiewicz's cabinet given vote of confidence,notclickbait +US army gives medical assistance to Iraq school,notclickbait +20 Badass Tattoos Inspired By Health And Wellness,clickbait +34 Of Your Favourite Bollywood Celebrities' First Tweets,clickbait +Read This Inspiring Post By Robin Williams' Daughter On Depression,clickbait +"The Cast Of ""Maze Runner"" Imitated Dylan O'Brien On The Red Carpet",clickbait +"Mia Farrow, Carole White testify in Charles Taylor's war crimes trial",notclickbait +Refugees From Region in Pakistan Trickling Home,notclickbait +"At the Open, Rain Tops the Leader Board",notclickbait +Football: Kežman goes to Madrid,notclickbait +Kung Fu star Carradine found dead in Bangkok hotel,notclickbait +"American disc jockey ""DJ AM"" dies at 36 of suspected drug overdose",notclickbait +Awesome Irn-Bru Cocktails,clickbait +Southern Minnesota plane crash kills eight,notclickbait +U.S. Captain Hears Pleas for Afghan Detainee,notclickbait +Toyota recalls up to 1.8 million automobiles,notclickbait +Can You Guess Why This Winchester Brother Is Crying,clickbait +Holy Shit This Guy Looks Exactly Like A Young Leonardo DiCaprio,clickbait +"This Movie Could Be The Next ""The Room""",clickbait +"In Minnesota, a Battle Without End for a Senate Seat",notclickbait +Volvo Ocean Race Comes to a Close,notclickbait +Kylie Jenner And Jenna Marbles Read Mean Tweets About Their Dogs And It Was Hilariously Mean,clickbait +Gay Men Asked Straight Men What They've Always Wanted To Know,clickbait +A New BlackBerry Curve Loaded With Features,notclickbait +US Senator Kennedy has brain tumor surgically removed,notclickbait +Here's How Bollywood Dressed Up For Halloween,clickbait +Can We Guess Your Personality Based On Your Favorite Things,clickbait +Banks Foreclose on Builders With Perfect Records,notclickbait +Sharapova knocked out of Wimbledon 2005 in semi-final,notclickbait +$8 Million Award in First Solo Tobacco Trial,notclickbait +NASA: Hopes raised for shuttle flights to resume soon,notclickbait +Which Classic Movie Should Your Intro To Film Essay Be About,clickbait +EU reaches budget deal,notclickbait +American-born terrorist gets 24 years,notclickbait +How Much Of A Night Person Are You,clickbait +"Closing the Roof on a New Era, for Better and Worse",notclickbait +19 Reasons The Nintendo 64 Is The Greatest Console Of All Time,clickbait +Cracks in Muzak Monolith as a Young Rival Grows,notclickbait +This Pastry Chef Matches His Treats To His Outfit And It Works,clickbait +"Norway Wins, While Russia Calls the Tune",notclickbait +"19 GIFs That Will Make You Say ""Ahhhhhhh""",clickbait +Convicted murderer Leonard Peltier again denied parole,notclickbait +People Keep Making Huge Facebook Chats With People With The Same Name,clickbait +Union Head Says Investigating Agents Is a Difficult Task,notclickbait +Fourteen dead in two attacks in Somalia,notclickbait +What Was The Worst Netflix And Chill You've Experienced,clickbait +Who Said It: Taylor Swift Or Dr. Phil,clickbait +"19 Life Lessons From ""500 Days Of Summer""",clickbait +New Orleans officials confiscating guns,notclickbait +Court legalizes same-sex marriage in New Brunswick,notclickbait +Are You More Like Ina Garten Or Giada De Laurentiis,clickbait +23 People Who Failed Spectacularly At Cooking,clickbait +Feist leads 2008 Juno Award winners,notclickbait +Conrail Plan May Be Model for G.M.-Chrysler Crisis,notclickbait +BT's Lalani smashes through the £1m salary barrier,notclickbait +"This ""Be Like Bill"" Meme Passive Aggressively Calls Out People's Social Media Habits",clickbait +Are You More DJ Khaled Or Miranda Sings,clickbait +I.B.M. Affirms Earnings Goal Despite Sales Slide,notclickbait +Grenade set off outside Tajik ministry,notclickbait +"Its Muscle Car Glory Faded, Pontiac Shrivels Up",notclickbait +We Know Your Favorite Dog Breed Based On Your Favorite Snack,clickbait +'Crown Fire' forces residents in Southern California to evacuate homes,notclickbait +18 Rotties Who Are Actually Just Big Teddy Bears,clickbait +"Thousands Of People Are Sharing This ""Humans Of Bombay"" Post About A Child With A Rare Genetic Disorder",clickbait +19 Fabulous Hacks To Make Your Shoes Look And Fit Perfectly Every Time,clickbait +Jada Pinkett Smith Posted A Baby Photo Of Will For His Birthday And It'll Make You Believe In Love,clickbait +11 Quotes From Harry Potter To Help You Cope With Loss,clickbait +Milk Scandal in China Yields Cash for Parents,notclickbait +South Korean scientists claim they have cloned pet dog,notclickbait +17 Textbooks That Forgot How To Be Textbooks,clickbait +17 Things All Girls With Too Much Hair Will Understand,clickbait +"Murdoch empire in crisis after newspaper closes: BSkyB bid halted, former editor arrested, anger at chief executive",notclickbait +This Guy Invented Extra Long Selfie Arms And It's The Weirdest Thing Ever,clickbait +"Thoughts I Had While Watching The Season 11 Premiere Of ""Keeping Up With The Kardashians",clickbait +Let These Hot Guys Help You Count Down The Days 'Til Christmas,clickbait +Another One Of Your Favorite Instagram Stars Is Quitting Social Media,clickbait +Social Networks Spread Defiance Online,notclickbait +"This Infographic Makes That All-Male ""Vanity Fair"" Photo Even Worse",clickbait +An Open Letter To Anyone Who's Ever Tweeted Anything Mean About The Rock,clickbait +UK experiences first widespread snow of 2009,notclickbait +Adults Sit On Santa's Lap For The First Time,clickbait +29 Things Non-Germans Find Bewildering About Germany,clickbait +9 Painfully True SEX-Pectations Vs Reality,clickbait +Denver Broncos player Kenny McKinley found dead aged 23,notclickbait +Day 4 Of BuzzFeed's 7-Day Clean Eating Challenge,clickbait +This Spoof Kids Book About Hipsters Is Hilarious,clickbait +13 Reasons Why Taylor Swift's 1989 World Tour Was The Best Part Of 2015,clickbait +Bomb Kills 7 Afghan Civilians at U.S. Base,notclickbait +Which Iconic Celebrity GIF Are You Based On Your Biggest 2015 Achievement,clickbait +34 Disturbing Photos From SantaCon That Will Ruin Your Christmas,clickbait +Republican Throwbacks Taking Aim at Court Pick,notclickbait +"Holding On to Wig, Dodgers Market Fallen Star",notclickbait +Stock markets worldwide continue to fall,notclickbait +"Are You More North, South, Or Central New Jersey",clickbait +"Worried About Flu, China Confines U.S. Students",notclickbait +Bringing Order to the Chaos of Notes,notclickbait +"Which ""Star Wars"" Spaceship Should You Fly",clickbait +There Are So Many More Penis Emojis Now,clickbait +Toronto transit union threatens strike for Monday,notclickbait +"As Giro Nears End, Contador Watch Starts",notclickbait +UK children's charities announce merger,notclickbait +17 Incredibly Important Life Lessons We Can Learn From Winona Ryder,clickbait +Answer These Four Questions And We'll Guess Your Gender,clickbait +Apple introduces new iPod with video playback capabilities,notclickbait +16 Snapchats That Prove DJ Khaled Is The Hero We All Need,clickbait +Emily Blunt Is Not A Fan Of John Krasinski's Hot New Bod,clickbait +Southwest Air to Begin Flying From Boston,notclickbait +Zeinab Sadiq Jaafar,notclickbait +13 Things About Language That Will Leave You Speechless,clickbait +"17 ""Home Alone"" Tweets That Will Make You Laugh Out Loud",clickbait +19 Painful Confessions About Being An Artist,clickbait +Israeli PM Ariel Sharon briefly opens eyes,notclickbait +We Know Your Zodiac Sign Based On The Things You Like,clickbait +"Football, American Style, Is Alive in France",notclickbait +Brittany Ray Scores 26 as Rutgers Beats Notre Dame,notclickbait +"All 78 ""Treehouse Of Horror"" Segments Ranked From Worst To Best",clickbait +"18 Insanely Cute Gifts You Should Buy ""Your Person"" Immediately",clickbait +How Much Of A Grammar Perfectionist Are You,clickbait +Police investigate Youtube video of two year old 'on ecstacy',notclickbait +How Well Do You See Color,clickbait +36 Crazy Gifts That Any Miyazaki Lover Will Go Nuts Over,clickbait +"Which ""How To Get Away With Murder"" Character Should Be Your Partner In Crime",clickbait +The Hardest Emoji Quiz You'll Ever Take,clickbait +"Matthew Edwards, honored Michigan police officer, shot and killed",notclickbait +24 Of The Most Motivational Celebrity Tweets Of All Time,clickbait +Bush nominates Alito to U.S. Supreme Court,notclickbait +"Central Michigan quarterback sets passing record, becomes finalist for award",notclickbait +People Are Emptying Out Giant Stuffed Bears And Climbing Inside,clickbait +Bolivian president announces legal action over Obama's 'crimes against humanity',notclickbait +North Korea Said to Detain U.S. Reporters,notclickbait +"If You Were Stranded On Mars, Which Vegetable Would You Eat",clickbait +India may rise as regional power,notclickbait +Drop Everything: Psy Released Two Insane New Music Videos,clickbait diff --git a/example/data/train.csv b/example/data/train.csv new file mode 100644 index 0000000..514580b --- /dev/null +++ b/example/data/train.csv @@ -0,0 +1,1401 @@ +headline,category +Stocks Barely Budge as Investors Await New Signals,notclickbait +25 Things You Will Only Understand If You Went To College In Texas,clickbait +54 Lady Gaga Lyrics For When You Need A Fire Instagram Caption,clickbait +BuzzFeed Crossword: CelebriBabies,clickbait +22 Pictures People Who Are Good Students Will Never Understand,clickbait +Can You Predict The Winners Of The 2016 Golden Globe Awards,clickbait +"White farmer who beat Mugabe in court over land seizures, dies",notclickbait +Volunteering At An Abortion Clinic Made Me Lose Patience With The Abortion Debate,clickbait +Here's Why You Shouldn't Care About How Damn Cold It Is,clickbait +Cricket: Scotland v Pakistan abandoned due to rain,notclickbait +Here Are All The Winners At The 2016 BAFTAs,clickbait +13 Very Honest Confessions From People Having A Quarter-Life Crisis,clickbait +The One Tip That Will Help Guys Dress Better For The Rest Of Their Lives,clickbait +Tennis: Andy Murray wins Montreal Masters 2009,notclickbait +Police evict Vestas protesters,notclickbait +Ingredient Costs Push Profit Lower at General Mills,notclickbait +"You Need To Watch 5,000 Ducklings Run Into A Pond Right Now",clickbait +Explosion Kills Afghan Police Chief and 3 Officers,notclickbait +15 Holiday Cocktails That Are Basically Dessert,clickbait +Liam Hemsworth Dressed Like A Panda And Carried Jennifer Lawrence's Purse Along The Great Wall Of China,clickbait +Reminder That Jennifer Lopez Is Not A Real Human,clickbait +"Thai PM barred from politics, three parties dissolved",notclickbait +21 Last-Minute Gifts That Are Actually Thoughtful,clickbait +"In Book, Radomski Talks About Dealing Drugs and Dealing With Mitchell",notclickbait +"Can You Match The ""BoJack Horseman"" Quote To The Character",clickbait +Can You Guess The Avril Lavigne Video From One Screenshot,clickbait +Pacific Rim braces for tsunami following major Chilean earthquake,notclickbait +Today's Good Nice Horoscope For All Humans,clickbait +16 Truths For People Who Get Major Anxiety When Texting,clickbait +13 Awkward Moments That Happen When You Get Braces,clickbait +Iran Stepping Up Effort to Quell Election Protest,notclickbait +Which Kardashian/Jenner Should Be Your BFF,clickbait +We Know Your Zodiac Sign Based On Your Favorite Dog Breeds,clickbait +Tight Mortgage Rules Exclude Even Good Risks,notclickbait +Here's How You Can Help Your Friend Recovering From An Eating Disorder,clickbait +Are You More Of A Broadway Stage Musical Or Movie Musical,clickbait +Monica Seles Among 4 Inducted Into Hall of Fame,notclickbait +"John Krasinski Wants To Clear Up That He And Jenna Fischer Were Never ""Genuinely In Love""",clickbait +13 Cocktails That'll Knock Santa's Boots Off,clickbait +"As Italy prepares for new government, shots fired near prime minister's office",notclickbait +Which Christmas Cookie Matches Your Personality,clickbait +First casualty of French riots reported,notclickbait +Canada's Dollaramas Are Raising Their $1 Prices And It's Bullsh*t,clickbait +"Planets Jupiter, Mercury and Mars line up, visible to naked eye",notclickbait +We Tried Stand-Up Comedy For The First Time And It Was Just Awful,clickbait +"Wikinews interviews Jon Greenspon, independent candidate for US President",notclickbait +War Hero in Vietnam Forces Government to Listen,notclickbait +Here's What Meg Cabot Learned About Situational Depression,clickbait +Can We Pickle That,clickbait +17 Hilarious Tumblr Posts That Will Make You Question Everything You Know About Language,clickbait +Elle May Have Just Changed The Women's Magazine Game With Its February Covers,clickbait +Are You Dead,clickbait +Ryan Curry Was Definitely The Cutest Baby Born In 2015,clickbait +Richard Gasquet Confirms Failed Drug Test but Disputes It,notclickbait +Miss California USA Will Keep Her Title,notclickbait +27 Things Everyone Obsessed With Ikea Will Understand,clickbait +Europe Looks to Africa for Solar Power,notclickbait +"In U.S., Steps Toward Industrial Policy in Autos",notclickbait +California company recalls fresh spinach over salmonella contamination,notclickbait +Los Angeles City Council to sue police officer accused of filing a false report,notclickbait +Here's Proof That Brining Your Turkey Is Stupid And Wrong,clickbait +Steel Industry: Tata buys Corus,notclickbait +Tour de France: Yellow jersey Rasmussen withdrawn,notclickbait +Can You Identify The Iconic Video Game Character From Just A Color Scheme,clickbait +"Hey Guys, Ted Mosby Is In India",clickbait +Energy Costs Lift Retail Sales and Producer Prices,notclickbait +Crisis Reshaping Wall St. as Stars Begin to Scatter,notclickbait +28 Tweets That Sum Up What It's Like To Grow Up In New Jersey,clickbait +"This Kid Stopped On His Way To Second Base To Tell His Baseball Coach ""I Love You""",clickbait +Which Cartoon Family Do You Belong In,clickbait +Obama (as TV Salesman) Pushes Home Refinancing,notclickbait +USC defeats Penn State in 95th Rose Bowl Game,notclickbait +Fans worldwide queue for new iPhone,notclickbait +US East Coast prepares for blizzard,notclickbait +Ross on Wye Friends win gold for garden,notclickbait +"If You Don't Know How Giraffes Sleep You Are Missing Out, My Friend",clickbait +Find Out What You Should Watch On Netflix This November,clickbait +Building in Barcelona damaged by explosion,notclickbait +Cities Deal With a Surge in Shantytowns,notclickbait +Panic! At The Disco's New Music Video Is The Creepiest Thing You'll See All Week,clickbait +This Guy Proposed With A Custom Super Mario Brothers-Style Video Game,clickbait +Final report blames London passenger jet crash on ice,notclickbait +"Italy Agrees to Take Migrants, Ending Standoff With Malta",notclickbait +Newcastle United's St. James' Park naming rights go up for sale,notclickbait +FBI says Jennifer Hudson's nephew found dead,notclickbait +"Which Character From ""Mob City"" Are You",clickbait +Phone hacking scandal prompts media review in Australia,notclickbait +Connecticut Serial Killer Competent- Waives Further Appeals,notclickbait +Warming oceans make it harder for fish to breathe,notclickbait +Gerrans Wins 14th Stage of Giro on Uphill Stamina,notclickbait +Co-Workers Surprise Each Other With Compliments,clickbait +This Dog Riding A Bike With A Helmet On Is The Cutest Damn Thing You'll See All Day,clickbait +Magazine Ad Pages Down Almost 26% in Quarter,notclickbait +10 Books That Will Get You In The Mood On Valentine's Day,clickbait +Mobile phones to help fight AIDS in Africa,notclickbait +Which Pop Artist/Producer Duo Are You,clickbait +"How Well Do You Remember Season 1 Of ""Friends""",clickbait +Professor Tied to Shootings Found Dead,notclickbait +We Know How Addicted You Are To Social Media Based On One Question,clickbait +Leftist Party Wins Salvadoran Vote,notclickbait +"17 Tweets About ""The Force Awakens"" That Will Make You Laugh Your Ass Off",clickbait +"The Fish Are Biting, and the Room Is Hopping",notclickbait +Missouri Edges Kansas With a Last-Second Shot,notclickbait +17 Tumblr Posts About Yoda Guaranteed To Make You Laugh,clickbait +Philippines prepares for new typhoon,notclickbait +"Do You Know These Secondary ""Game Of Thrones"" Characters",clickbait +This Is For Everyone From Upstate New York,clickbait +Australian Minister 'leaks' draft of anti-terror bill,notclickbait +13 Terrifying Urban Legends That Will Make You Wish You Hadn't Read This,clickbait +This Veterinarian Ate With A Neglected Dog In A Cage To Make Her Feel Comfortable,clickbait +"Schröder gives up German chancellorship ambitions, makes way for Merkel",notclickbait +Silverjet ceases operations and enters administration,notclickbait +"Laws allowing same sex marriage in Washington, D.C. go into effect",notclickbait +10 Go-Go Remakes You Absolutely Must Hear,clickbait +When You're Single For The Holidays,clickbait +17 Dads Who Are Dad AF,clickbait +How To Turn Classic Thanksgiving Foods Into Boozy Cocktails,clickbait +China's economy surpasses Japan's in second quarter,notclickbait +Which Disney Character Sparked Your Sexual Awakening,clickbait +A Top Sunni Survives an Attack in Iraq,notclickbait +33 Weird Things You'll Find In Every British Person's Garage,clickbait +David Beckham Just Got Romeo The Best Birthday Present,clickbait +Sarkozy Backs Drive to Eliminate the Burqa,notclickbait +A Whiff of Controversy and South African Wines,notclickbait +Sailor Finishes Ocean Race After 121 Days at Sea,notclickbait +"Apple to give free cases, refunds to iPhone 4 owners",notclickbait +12 Incredible Photos Of An Icelandic Glacier That Will Make You Want To Go There Right Now,clickbait +"Small plane makes crash landing in Massachusetts, USA",notclickbait +McCain and Obama participate in Saddleback Church forum,notclickbait +"Do You Know The Lyrics To ""How Deep Is Your Love"" By Calvin Harris And Disciples",clickbait +We Know Your Favorite Disney Movie Based On Your Favorite Dog,clickbait +Move to Redefine New England Fishing,notclickbait +Heidi Klum Transformed Into Jessica Rabbit To Prove She's The Queen Of Halloween,clickbait +California state legislature passes same-sex marriage bill,notclickbait +"The Goddam Internet Did It Again With This Lip-Dub Of Olivia From ""The Bachelor""",clickbait +How Creepy Was Terrence Howard's Kiss At The Emmys,clickbait +"According to UN report, bribes cost Afghans US$2.5 billion per year",notclickbait +Pope Benedict XVI visit to the United States begins,notclickbait +"Delta flight makes emergency landing at JFK, no injuries",notclickbait +Toyota quits Formula One,notclickbait +"Is the Beautiful Game's Ethic Money, Glory or Work?",notclickbait +18 Things Japan Has That The Rest Of The World Desperately Needs,clickbait +Columbia Moves a Game Behind First Place in the Ivy,notclickbait +"Emo Kylo Ren Is Your New Favorite ""Star Wars"" Parody Account",clickbait +19 Of The Most Hilarious Tweets About Ryan Gosling,clickbait +We Need To Talk About Selena Gomez At The Victoria's Secret Fashion Show,clickbait +"14 ""Hunger Games"" Questions That Are Impossible To Answer",clickbait +India's ruling Congress party leads in three key state polls,notclickbait +"Over 100 Shiite rebels killed in Yemen, government says",notclickbait +Lebanon: UN resolution won't end conflict,notclickbait +Britney Spears 9 Years Ago Vs. Today,clickbait +20 Cereal Recipes to Stress-Eat Between New Episodes of Serial,clickbait +13 Annoying Little Problems Trans Women Didn't Expect After Transition,clickbait +Prosecutors begin NY State Sen. Hiram Monserrate felony assault case,notclickbait +"Redskins Sign Haynesworth, but They Pay Top Dollar",notclickbait +"Subway train derails in Washington, D.C.",notclickbait +Lenders Pledge $31 Billion for Eastern and Central Europe,notclickbait +Senior U.S. Diplomat Visits Myanmar,notclickbait +Russia's Nomadic Reindeer Herders Face The Future,clickbait +New York Doctors Race to Abide by In-Office Surgery Law,notclickbait +Germany's July unemployment rate falls slightly,notclickbait +This One Question Will Tell You If You're A Real Woman,clickbait +There's A New Taylor Swift Meme Sweeping Tumblr,clickbait +"Hospitals declare ""code red"" in Wellington, New Zealand",notclickbait +Meet The Girl Who Buttchugged Cough Syrup,clickbait +21 Pictures About Work Guaranteed To Make You Laugh,clickbait +People Met Pet Rats For The First Time And Lost Their Damn Minds,clickbait +"Marnie Piper Is Returning To The Town From ""Halloweentown""",clickbait +24 Things You'll Understand If You Think Working Out Is The Worst,clickbait +"Covered Up, and Harassed, in Cairo",notclickbait +Greece wildfires force thousands to evacuate,notclickbait +I'm Asexual But I'm Not,clickbait +Malin Akerman Plays The World's Hardest Game Of Would You Rather,clickbait +21 Ways To Cover Your Home With Pictures Of Your Own Cat,clickbait +16 Times Tumblr Made You Think Too Hard,clickbait +U.N. Report Lays Out Options for an Oil-Rich Iraqi Region,notclickbait +Alcoa Lost $1.19 Billion as 4th-Quarter Sales Fell,notclickbait +Singer Britney Spears gives birth to a healthy boy,notclickbait +Drake Shared A Sweet Instagram Post Thanking Rihanna For His Only No. 1 Songs,clickbait +Study of soft cheese wins oddest book title award,notclickbait +"Which Taylor Swift ""1989 World Tour"" Guest Are You Based One Question",clickbait +Mozart In The Jungle Should Be The Next Show You Binge-Watch,clickbait +"Last Gaza settlement cleared, West Bank towns prepare to resist",notclickbait +15 Awards For Anyone Who Definitely Has What's Going Around,clickbait +Biology and Technology Intersect to Create Art,notclickbait +Washington Churches Eye the Obamas,notclickbait +Beijing Investigates Transplants for Tourists,notclickbait +"Scuderi, an Unlikely Savior, Emerges for the Penguins",notclickbait +China successfully launches Shenzhou VI manned rocket,notclickbait +Are You Really A Mermaid,clickbait +35 Foods That Pretty Much Sum Up Your Childhood,clickbait +"If ""Harry Potter"" Characters Were In ""Game Of Thrones""",clickbait +21 Wedding Photos You'll Want To Pin Immediately,clickbait +Tell Us Your Stories About Sexism In College,clickbait +The Struggle Of Being Mixed Race,clickbait +38 Times Tom Hardy Melted Your Heart With His Love For Dogs,clickbait +Australia proposes new anti-terror laws,notclickbait +Surprising Beauty Hacks Using Coffee Grounds,clickbait +22 Struggles Of People With Zero Athletic Ability,clickbait +Oil prices fall as reserves are released,notclickbait +NASA's Phoenix spacecraft lands safely on Mars,notclickbait +Which Zodiac Sign Do You Think Fits You Best,clickbait +Volcanic activity expands McDonald Island off Australia,notclickbait +A Reminder That Maru Is Still King Of The Internet,clickbait +What Adorable Nickname Should You Give Your Dog,clickbait +What's Your Favorite Book With A Great LGBT Lead Character,clickbait +Bulusan Volcano releases some ash,notclickbait +Russian oil depot burning after explosion,notclickbait +NASA: Old Arctic sea ice continues to melt,notclickbait +Study estimates first human HIV infection 100 years ago,notclickbait +"Dennis Ritchie, C programming language creator, dies aged 70",notclickbait +I Bet You Can't Identify Each Of These Disney Movies,clickbait +Israel undergoes major emergency drill,notclickbait +Emojis: What The Fuck Do They Mean,clickbait +Australia commits more troops to Afghanistan,notclickbait +Former Texas nurse charged with murder for allegedly injecting bleach into patients,notclickbait +You Probably Don't Know The Real Reason Chinese Women Bound Their Feet,clickbait +Canberrans spend Easter outside: in pictures,notclickbait +Unions battle in Ohio over hospital workers,notclickbait +The Word Of The Day Is Thanks To Queen Amy Poehler,clickbait +9 Celebrities As Zombies And Some Undead Fiction,clickbait +Some People Think Gigi Hadid Kinda Looks Like Fergie,clickbait +Adult Survivors Of Child Sexual Abuse Are Speaking Up In This New Video Series,clickbait +Wikinews interviews author and filmmaker Peter John Ross,notclickbait +IAEA: North Korea to begin shutting down nuclear reactor next week,notclickbait +Can We Guess Which Kardashian/Jenner Sister Is Your Favorite Based On Three Questions,clickbait +Let's All Take A Moment To Appreciate The Uncontested Queen That Is Julie Andrews,clickbait +"Dam Bursts in Indonesia, Killing Dozens in Flood",notclickbait +Portuguese government resigns,notclickbait +Libyan foreign minister defects to the UK,notclickbait +How Well Can You Really Match Colors,clickbait +Clashes in France after anti-condom speech by Pope,notclickbait +How ~Deep~ Are You,clickbait +Oak Park and neighbors to study Les Golden's intergovernmental cooperation proposals,notclickbait +This Is The Best Way To Netflix And Chill,clickbait +Can We Guess What You Look Like From These 10 Questions,clickbait +24 Tweets That Will Make Every Nurse Laugh Out Loud,clickbait +Launch of Space Shuttle Discovery further delayed,notclickbait +All 2500 runners in Lake District race accounted for after flood,notclickbait +Israeli Defense Force admits to targeting media center in Gaza City airstrike,notclickbait +Australia First begins revival campaign,notclickbait +Do You Know The Decade In Which These Movies Came Out,clickbait +19 Things Anyone Who's Best Friends With Their Mum As An Adult Will Understand,clickbait +Jennifer Lawrence And Amy Schumer Together At The Golden Globes Was Perfect,clickbait +Google Chief for Charity Steps Down on Revamp,notclickbait +Can You Name These Adorable Instagram-Famous Pets,clickbait +"Do You Still Know The Original ""Sister, Sister"" Theme Song",clickbait +Indians Are Protesting Child Marriage With The Hashtag #StrengthToSayNo,clickbait +Many G.M. Plants Will Take the Summer Off,notclickbait +Epiphanny Prince Scores 24 Points as Rutgers Tops West Virginia,notclickbait +Sheffield tram-train project back on the rails,notclickbait +Glastonbury headliners announced,notclickbait +What's The Best Haunted House You've Ever Been To In The U.S,clickbait +Two dead after building collapse in Kenya,notclickbait +26 Products For People Who Are Completely Obsessed With Snakes,clickbait +Investigators discover hole in fuel tank of burnt-out China Airlines jet,notclickbait +We Need To Talk About Gigi Hadid For Like 10 Seconds,clickbait +Two new cases of West Nile Virus reported in Indiana,notclickbait +16 Things Dudes Need To Stop Wearing In 2016,clickbait +Peer-to-peer file-sharing user numbers still growing,notclickbait +Which Early 2000s Disney Show Matches Your Zodiac,clickbait +Man arrested after crashing motorized bar stool while drunk,notclickbait +"Zoo elephants live shorter lives than their wild counterparts, report warns",notclickbait +Ferry collision on the river Mersey,notclickbait +15 Vibrator Horror Stories That'll Make You Cringe,clickbait +British man survives artificial heart transplant,notclickbait +Davie Shipyard sale cleared by Québec court,notclickbait +US indicts eleven alleged pirates from Somalia,notclickbait +Lupita Nyong'o Had The Perfect Response To Lack Of Racial Diversity At The Oscars,clickbait +Meet The 4-Year-Old Twins Who Are Taking The Modeling World By Storm,clickbait +Southern California hit by 5.1 earthquake,notclickbait +People Are Using The Hashtag #BurritoSelfie And It Is As Glorious As You'd Imagine,clickbait +Executions Debated as Missouri Plans One,notclickbait +Virginians melee at used Apple iBook sale,notclickbait +11 Classic Movie Lines From Alan Rickman,clickbait +14 Duets That Prove Two Voices Are Better Than One,clickbait +UN holding recruitment exams in under-represented countries,notclickbait +Bank of America to Receive Additional $20 Billion,notclickbait +60 reported dead in Congo train crash,notclickbait +Malicious Software Is Revised,notclickbait +British comedian Eddie Izzard completes 43 marathons in 51 days,notclickbait +"Go Home Everyone, Sir Patrick Stewart Won 2015",clickbait +Duke Outlasts Binghamton,notclickbait +Stolen Utahraptor recovered in Australian Capital Territory,notclickbait +Zidane apologises for headbutt,notclickbait +Texas Tech coach may move to University of Miami,notclickbait +Bomb blasts kill several in Iran,notclickbait +Would You Time Travel To The Past Or Future,clickbait +This Touching Photo Of A Bride And Her Service Dog Is Going Viral,clickbait +19 Secrets Wedding Caterers Won't Tell You,clickbait +BBC drops programmes as third of staff join strike,notclickbait +"Definitive Proof ""Titanic"" And ""Peter Pan"" Are Almost The Same Movie",clickbait +Country Star Jason Aldean Wore Blackface On Halloween To Dress Up As Lil Wayne,clickbait +Couples Find Out If Their Stories Of How They Met Add Up,clickbait +Former Astronaut Bolden to Be Interviewed for Top NASA Job,notclickbait +Chris Langham found guilty of downloading child pornography,notclickbait +27 Tweets About Starbucks That'll Actually Make You Laugh,clickbait +Little Rock Nine member Jefferson Thomas dies aged 67,notclickbait +What Kind Of Person Are You Really,clickbait +Arrest made in Newark Airport security breach,notclickbait +Montreal lab questions ethics of recent EPO doping claims against Lance Armstrong,notclickbait +15 Surprising Actors Who Unknowingly Sparked My Sexual Awakening,clickbait +34 Cosplayers Who Slayed MCM London Comic Con 2015,clickbait +"As Giants Break Camp, Coughlin Uses Lakers as Motivation",notclickbait +How To Make Sheet Pan Salmon With Crispy Kale,clickbait +Here's What The Universal Monsters Would Look Like Today,clickbait +This Color Test Will Determine Your Guilty Pleasure,clickbait +"22 Hilarious Pictures Only ""The Walking Dead"" Fans Will Understand",clickbait +New Zealand dollar hits new high versus the United States dollar,notclickbait +Indian Army celebrates Victory Day on 35th Anniversary of Bangladeshi Liberation,notclickbait +Large earthquake hits central China,notclickbait +Scientists proudly display oldest crystal on Earth,notclickbait +Afghanistan general Stanley McChrystal cleared of wrongdoing,notclickbait +Confessions To The One That Got Away,clickbait +We Know Your Favorite Music Genre Based On Your Zodiac Sign,clickbait +How A Decade Of Sobriety Finally Taught Me To Drink,clickbait +Are These Women More Than Just Friends,clickbait +OPEC Backs Away From New Output Cuts,notclickbait +Guard of Key Witness Killed Before Terror Trial in Greece,notclickbait +Can Americans Pass The UK Driving Test,clickbait +ABC to Merge 2 TV Units to Streamline and Cut Costs,notclickbait +Amy Schumer's Press Room Interview After She Won Her Emmy Is Iconic,clickbait +Happy Gilmore Was On to Something,notclickbait +Everything You Need To Know About Empress Of,clickbait +"We Know Your ""Harry Potter"" Trick-Or-Treating Partner Based On Your Birth Month",clickbait +Body of Rosa Parks to lie in honor at U.S. Capitol,notclickbait +2015 Was The Year Shibas Proved They Were More Than A Meme,clickbait +This Video Of A Grandpa Singing To His 93-Year-Old Dying Wife Will Give You All The Feels,clickbait +Moore Scores 40 and Sets Marks for No. 1 UConn,notclickbait +Do You Actually Remember These Disney Channel Show Theme Songs,clickbait +For Anyone Who's Had A Huge Fight With Family Over Data,clickbait +"After a Pause, Wall Street Pay Bounces Back",notclickbait +New Zealand's South Island and southern North Island struck by storms,notclickbait +"In China, G.M. Remains a Powerful Player",notclickbait +"Knicks Rout Grizzles, Extending Win Streak to Three Games",notclickbait +Police social media frenzy threatens to prejudice alleged child killer trial,notclickbait +"In Bleak Afghan Outpost, Troops Slog On",notclickbait +"In Minnesota, Another Bid for a Recount",notclickbait +Ashes Teams Gasp for Breath,notclickbait +Santas Guess Bizarre Candy Cane Flavors,clickbait +38 Genius Ways To Save Money On Travel In 2016,clickbait +22 Confessions From Erotica Writers,clickbait +"Which ""The Emperor's New Groove"" Character Are You",clickbait +China executes nine for ethnic riots,notclickbait +Two Prime Ministers banned from Fiji,notclickbait +"Afghan Girls, Scarred by Acid, Defy Terror, Embracing School",notclickbait +YouTube access returns to Thailand,notclickbait +This Is What Happens When You And Your Best Friend Have No Boundaries,clickbait +20-Minute Chicken Recipes To Add Your Arsenal,clickbait +Democrat staffers obtained Maryland lt. governor's credit report illegally,notclickbait +"Rory Williams Has Pretty Much Graduated From Companion To Doctor On ""Legends Of Tomorrow""",clickbait +2015's Biggest Memes Are Now Emojis,clickbait +Wikinews interviews Daylight Saving for South East Queensland Party about the upcoming Queensland State election,notclickbait +Are You More D.J. Tanner Or DJ Khaled,clickbait +Red Cross asks for more volunteers,notclickbait +WHO: H1N1 influenza virus still a pandemic,notclickbait +22 Things Scottish People Have Been Trying To Tell You For Years,clickbait +Leonardo DiCaprio's Face When Lady Gaga Walked By Him To Accept Her Award Is Everything,clickbait +"People Are Upset That Rey Is Missing From ""Star Wars: The Force Awakens"" Toy Sets",clickbait +France Opens First Military Bases in the Gulf,notclickbait +Bomb explosion in Pakistani market kills 49,notclickbait +Pediatrician in Molesting Case Agrees to Stop Practicing Medicine,notclickbait +Wider Drug War Threatens Colombian Indians,notclickbait +'Bloody Sunday Inquiry' publishes report into British Army killing of activists in Northern Ireland,notclickbait +'Pregnancy pact' grabs international attention for small Massachusetts town,notclickbait +Second man charged in Lee Rigby murder case,notclickbait +Here Are 18 Romantic AF Movies & Shows To Stream On Valentine's Day,clickbait +We Know Which Fan Fiction You Should Read Based On Your Fandom,clickbait +3 Refugees Living In The UK Cook Their Favourite Meals From Home,clickbait +San Francisco anti-G8 rally turns violent,notclickbait +2008 Taiwan Tourism Exposition to encourage tourism industry after Sichuan earthquake,notclickbait +"26 Jokes Guaranteed To Make ""Star Wars"" Fans Laugh Every Time",clickbait +KJ Noons stripped of EliteXC Lightweight Title,notclickbait +The Mona Lisa Just Turned Down Cara Delevingne For A Date,clickbait +Lakewood Police & SWAT team end standoff with tear gas,notclickbait +"Meet Gar Ryness, the Batting Stance Guy",notclickbait +Broadcaster Paul Harvey dies at age 90,notclickbait +Mobile ringtone tops the UK singles chart,notclickbait +"From Blog to Print, Laughing All the Way to the Bank",notclickbait +Health expert: Swine flu outbreak exaggerated by pharmaceutical companies for profits,notclickbait +17 Gifts For People Whose Best Friend Is A Pomeranian,clickbait +Southern Ocean whale slaughter to resume,notclickbait +African nations gather to support a ban on cluster bombs,notclickbait +Native Americans Confessed How They Feel About America And It Got Real,clickbait +Libertarian National Committee in fierce deadlock over how to address growing Bob Barr controversies,notclickbait +"The New ""Little Mermaid"" Is Going To Be Blonde And People Aren't Happy",clickbait +33 Love Songs From The Early '00s You'll Never Forget,clickbait +Social Media Vs. Real Life,clickbait +"Arturo Gatti, Former Boxing Champion, Found Dead in Hotel Room",notclickbait +18 Shocking Big Dick Confessions,clickbait +Nepal's royal palace now a public museum,notclickbait +Which Non-Prince Disney Guy Is Your Soulmate,clickbait +When You Hate Your BF's Mustache,clickbait +Bomb explosions kill several people in central Baghdad,notclickbait +United States bank receives death threats,notclickbait +11 Things You've Definitely Googled Before,clickbait +What's Your Subconscious Obsessed With,clickbait +"This Is What The Cast Of ""Made In Chelsea"" Would Look Like As Dogs",clickbait +"Are You More ""In The Heights"" Or ""Hamilton""",clickbait +British jazz musician John Dankworth dies aged 82,notclickbait +Secret memos reveal Bush administration endorsed enhanced interrogation techniques,notclickbait +17 Times The Mom On Clarissa Was A Goddamn Style Icon,clickbait +Did You Ruin Everything,clickbait +Over 1.5m apply for British Live 8 tickets,notclickbait +Here's How Orphan Black Fans Are Dealing With Tatiana Maslany's Emmy Loss,clickbait +Pakistan and Taliban Battling for Key City,notclickbait +Mariah Carey's Insane Engagement Ring Belongs In A History Textbook,clickbait +22 Questions For The Monsters Who Don't Like Dogs,clickbait +Mubarak summoned by Egyptian prosecutors to face allegations of killings and corruption,notclickbait +These Straight Men Asked Gay Men Questions About Life And Sex,clickbait +"Time magazine's 2006 Person of the Year is ""you""",notclickbait +15 TV Shows And Movies That Came Back From The Dead In 2015,clickbait +"A Good Run for Mutual Funds, but Questions Remain",notclickbait +A World Cup Qualifier. Cool.,notclickbait +Everybody Hates Mark From Match.com,clickbait +Coordinated series of bombs kills at least 80 in India,notclickbait +Aviation experts suggest Air France Flight 447 broke up in midair,notclickbait +Two Twin Brothers Separated Since World War II Have Finally Been Reunited,clickbait +Are You More Matt Damon Or Ben Affleck,clickbait +Can You Identify The 2015 Music Video By Its YouTube Comment,clickbait +Russian passenger airliner makes safe landing in Moscow after concerns of gear damage,notclickbait +Zimbabwe cancels education year for 4.5 million after political and economic troubles,notclickbait +Yankees Grab a Share of First Place as Everyone Lends a Hand,notclickbait +19 Things You Can Only Say During Sex If You're Emo,clickbait +18 Pictures That Are Way Too Real For People Who Shed A Lot,clickbait +Listen To Women Talk Honestly About Loving Their Bodies,clickbait +Jennifer Lawrence's New Movie Is Basically Just A Bunch Of GIFs Of Her In Search Of A Movie,clickbait +"Patient in Buckinghamshire hospital was treated in toilet, inquiry hears",notclickbait +India's Telecom Authorities Just Called Mark Zuckerberg's Bluff About Free Basics,clickbait +Automakers Seek $14 Billion More in Aid,notclickbait +11 Struggles All Hot Sauce Lovers Know,clickbait +Employment Contracts Are Now Viewed as Rewritable,notclickbait +This Video Is For Every Girl Who Can't Do Makeup,clickbait +British charities form fund recovery group,notclickbait +EU's human rights court endorses Turkish headscarf ban,notclickbait +Scientists create micro-battery using 3D printing,notclickbait +Caitlyn Jenner Totally Slayed It At The Point Foundation's Annual Voices On Point Gala,clickbait +18 Cats Who Don't Believe In Books,clickbait +Goldman Settles Subprime Complaint in Massachusetts,notclickbait +Here's What You Should Actually Do When You Have A Crush,clickbait +Which Video Game Princess Are You,clickbait +15 Purrfect Gifts To Honor The Crazy Cat Gentlemen,clickbait +18 Pugs Who Forgot Very Important Things,clickbait +Amy Schumer And Amy Poehler Were Perfect Together At The Emmys,clickbait +This Is What Porn Sets Look Like When People Aren't Having Sex In Them,clickbait +"Asian earthquake toll nears 60,000",notclickbait +"As Detroit Struggles, Foundations Shift Mission",notclickbait +"How Obsessed With ""Game Of Thrones"" Are You",clickbait +MySpace to display AMBER Alerts,notclickbait +Graner found guilty of mistreatment at Abu Ghraib,notclickbait +Jenny Li and Thomas Dold win the 2008 Taipei 101 Run Up,notclickbait +18 Times Dylan Sprouse Was The Absolute Best Person On Twitter,clickbait +North Carolina Beats Radford in First Round Without Lawson,notclickbait +21 Recipes That Will Completely Transform Boring Leftovers,clickbait +Geffen Said to Be Still Interested in The New York Times,notclickbait +16 Adults Reveal Their Hilarious Stories About The Last Time They Peed Themselves,clickbait +"Which ""Harry Potter"" Villain Are You Based On Your Zodiac",clickbait +21 Signs You're Addicted To Your Goddamn iPhone,clickbait +Nuclear leaks after Japan quake are worse than first reported,notclickbait +23 Insanely Cozy Robes To Spend Your Weekends In,clickbait +"Are You More Dipper Or Mabel From ""Gravity Falls""",clickbait +16 DJ Khaled Quotes That'll Motivate The Shit Out Of You,clickbait +Ousted Honduran president Zelaya says coup talks suspended,notclickbait +"Murdoch drops BSkyB bid amid public, political pressure",notclickbait +28 Picture Tweets Guaranteed To Make Absolutely Everyone Laugh,clickbait +"Voluntary student unionism bill passes Australian House of Representatives, enters Senate",notclickbait +"Series of earthquakes hit Korea, Indonesia, Turkey and Norway",notclickbait +Opera singer Pavarotti in serious condition,notclickbait +France Looks to Help Airbus With Loan Guarantees,notclickbait +Russia expels four UK diplomats,notclickbait +Viacom Profit Falls 69% as Revenue Remains Flat,notclickbait +Same-sex marriage reaffirmed in Canada,notclickbait +"Rangers Pick Up Where They Left Off, With a Victory",notclickbait +Everything You Need To Know About Floating Iceberg Hot Chocolate,clickbait +Survivors of High-Seas Drama Return to France,notclickbait +Wikimedia Commons celebrates first anniversary,notclickbait +"Explosion at new shopping centre in Bath, United Kingdom",notclickbait +"35 Questions We Want The ""Gilmore Girls"" Revival To Answer",clickbait +Foxconn under pressure after tenth employee suicide this year,notclickbait +Sondhi may face legal action from Thai Rak Thai party,notclickbait +Proton rocket fails to launch AMC-14 satellite,notclickbait +15 Struggles Everyone Who Has A Celebrity Crush Knows To Be True,clickbait +30 people die in bombing in northwestern Pakistan,notclickbait +"Clear Channel Plans to Trim 1,850 Jobs",notclickbait +Can We Guess Your Personality Based On Your Resting Bitchface,clickbait +Chinese version of Wikinews gets blocked in China,notclickbait +Can You Pass This Gross-Ass Bodily Fluid Quiz,clickbait +"Man dies in North Uist, Outer Hebrides after being hit by car",notclickbait +28 Profoundly Beautiful Quotes About Life And Death,clickbait +"Nortel, Once a Telecom Giant, Files for Bankruptcy",notclickbait +18 Deliciously Decadent Lemon Desserts To Die For,clickbait +Can You Get A Perfect Score On This Ron And Hermione Quiz,clickbait +IMF head remains in New York prison; charged over alleged hotel sex attack,notclickbait +The 29 Most WTF Moments From The 2015 MTV EMAs,clickbait +Watch People Struggle Trying To Guess Urban Dictionary Terms,clickbait +"Let's Be Honest, GOP Candidate Lindsey Graham Has Dazzlingly Beautiful Eyes",clickbait +American war deserter given stay of deportation in Canada,notclickbait +"How ""The Martian"" Went From A Best-Selling Novel To A Blockbuster Film",clickbait +"Do You Know If These Food ""Facts"" Are True Or False",clickbait +Australian wins 2005 World Series of Poker,notclickbait +"77 names added to fallen journalist memorial in Washington, D.C.",notclickbait +"Which Of Scully From ""X-Files"" Outfits Are You Based On Your Star Sign",clickbait +Restrictions Are Upheld for Executives in OxyContin Case,notclickbait +21 Life-Changing Products That Can Actually Make Your Skin Better,clickbait +"6.2 magnitude earthquake strikes Afghanistan, Pakistan",notclickbait +Rio de Janeiro to host 2016 Olympics,notclickbait +Evo Morales hoaxed by a Spanish Church-owned radio station,notclickbait +"Two firefighters killed in deadly blaze in Buffalo, New York",notclickbait +17 Drag Queen Transformations That Will Blow Your Mind,clickbait +The Simpsons episode to premiere on Sky1,notclickbait +Report finds Canberra and Northern Territory have most expensive cocaine in Australia,notclickbait +"Acquired in J. J. Putz Trade, Sean Green Also Seen as Valuable to Mets",notclickbait +We Know What Funny Dog GIF You Need To See Based On Your Zodiac,clickbait +The Full Story Of How 2015 Was The Year Humanity Lost All Hope,clickbait +Navy helping New Orleans pets,notclickbait +You Can Personalize Your Own Adult Coloring Book With Your Name On It,clickbait +Confirmed bird flu death in Nigeria,notclickbait +Canada Drops W.T.O. Complaint Against Europe,notclickbait +21 Hilarious Tweets That Prove Tyra Is The Most Relatable Supermodel Ever,clickbait +What Do You Want To Know About Tyler Ford,clickbait +"Pittsburgh Pirates defeat Philadelphia Phillies, 3-0",notclickbait +Report from Royal International Air Tattoo 2009,notclickbait +Are You More Furiosa Or Rey,clickbait +"Historic gym the site of Benet Academy, Illinois victory over Oswego",notclickbait +"Mozambican opposition rejects election results, calls for another vote",notclickbait +LaShawn Merritt and Sanya Richards Cruise to U.S. Titles in 400 meters,notclickbait +Exports Plunge in China,notclickbait +Last Minute Halloween Costume Ideas For Your Pet,clickbait +"Microsoft Office dropped by Massachusetts, USA",notclickbait +UConn Stifles Louisville In a Flash,notclickbait +18 Gordon Ramsay Insults All Servers Really Fucking Need,clickbait +There's A Weird Geographical Quirk About Sweden's Match Against Denmark This Week,clickbait +Second Claim of Paternity for President of Paraguay,notclickbait +20 Texts That Require No Response,clickbait +Avalanche in Canadian Rocky Mountains kills two,notclickbait +"Estonia Convicts Hermann Simm, Former Security Official, of Treason",notclickbait +Robot Zoe detects life in Atacama Desert,notclickbait +"2 Churches, Black and White, See Inaugural Hope",notclickbait +Watch This Woman Surprise Her Girlfriend With The News She Will Be Her Kidney Donor,clickbait +"Dead children found in car in Sussex, UK",notclickbait +15 Delectable Overnight Oats That Are Totally Worth Getting Out Of Bed For,clickbait +South African sprinter Philip Rabinowitz dies at age 104,notclickbait +Gannett Reports a 60% Fall in Quarterly Profit,notclickbait +"Google doubles Gmail storage, adds text formatting",notclickbait +Are You More Tyler Joseph Or Josh Dun From Twenty One Pilots,clickbait +National Guard mobilized in Kentucky ice storm aftermath,notclickbait +40 Thoughts Everyone Has When It Starts Raining Outside,clickbait +This Toddler's Instagram Photos Are Guaranteed To Make You Feel,clickbait +16 GIFs That Will Actually Help You In Maths Class,clickbait +CW Adds Shows to Text About,notclickbait +17 Meatballs Having An Identity Crisis,clickbait +I Keep Getting Mistaken For The Mayor Of Ottawa On Twitter,clickbait +This Is What Happens When A Blizzard Strikes On Your Wedding Day,clickbait +"How Brave Are You, Actually",clickbait +One dead after small plane crashes near Phoenix,notclickbait +Major storms batter Europe,notclickbait +Next Big Film Has a Premiere in Your Living Room,notclickbait +8 Things To Help You Talk To Your Family About Your Eating Disorder,clickbait +Agency Skeptical of Internet Privacy Policies,notclickbait +The Hardest Tube Station Quiz You'll Take Today,clickbait +Gwendoline Christie Showed Up To The Emmys Looking Like A Literal Goddess,clickbait +Can You Match The Quote To The Kardashian Who Said It,clickbait +What Should Your Career Actually Be,clickbait +Isn't She Lovely,clickbait +Which 2006 Hit Single Are You Based On Your Star Sign,clickbait +"27 Funniest Moments From ""Whose Line Is It Anyway?""",clickbait +Kyrgyzstan votes on referendum for new constitution,notclickbait +"Stephen Colbert Went Full ""Hunger Games"" Last Night",clickbait +A.M.A. Opposes Government-Sponsored Health Plan,notclickbait +Wall Street Stumbles as the Day Nears an End,notclickbait +Sex Predator Accusations Shake a Wisconsin Town,notclickbait +17 Things You Should Know Before Going Vegan,clickbait +Obama budget calls for record US deficit,notclickbait +"One killed by suicide bomber in southern Russia, five wounded",notclickbait +Afghan Insurgents Expand Their Use of Increasingly Sophisticated Homemade Bombs,notclickbait +WHO: Polio reemerging in Africa,notclickbait +Service Warranties and the Cost of Fun,notclickbait +China threatens to take action over US-Taiwan deal,notclickbait +Mersey hospital takeover by the Australian government,notclickbait +21 Signs That Prove Booksellers Are The Absolute Best,clickbait +These High School Students Just Shut It Down With This Epic Lip Dub Video,clickbait +"Shut It All Down, This Is The Finest Moment In Human History",clickbait +"N.C.A.A. Drought Over, Syracuse Moves to Second Round",notclickbait +Officer dies after accident in President Bush's motorcade,notclickbait +Can You Make It Through This Simple Yes Or No Quiz About Movie Quotes,clickbait +Slow Cooker Mexican Dishes,clickbait +Is 2016 The Year You Stop Being Single,clickbait +Thai Chef Breaks Down What Makes A Good Pad Thai,clickbait +When Your Friend Is A Coffee Snob,clickbait +When Your Mom Adds You On Facebook,clickbait +Priyanka Chopra Just Became The First South-Asian Actress To Win A People's Choice Award,clickbait +South Koreans Express Fatigue With a Recalcitrant North,notclickbait +This Doritos Ranch Queso Is The Drunk Snack You Deserve,clickbait +Lebanon car bombings kill dozens outside mosques,notclickbait +LeBron James Helps Cavaliers Complete Sweep of Hawks,notclickbait +Financial crisis strikes charity in Poland,notclickbait +Sorry But Lorelai Gilmore Is Kind Of The Worst,clickbait +UConn Men Overpower Texas A&M to Reach Round of 16,notclickbait +22 Sentences That Are Way Too Real For People Who Grew Up With Strict Parents,clickbait +Reese Witherspoon Just Called Out Hollywood Ageism And Sexism And It Was Beautiful,clickbait +What Type Of Thanksgiving Turkey Should You Make,clickbait +"Bloggers cite rumors of US ""secret war"" with Iran and Syria",notclickbait +Can You Identify The Miyazaki Character By Just Their Eyes,clickbait +Can You Really Tell The Difference Between Elena Gilbert And Katherine Pierce,clickbait +An iPhone Upgrade to Counter the Naysayers,notclickbait +Senator Obama's passport records breached in January 2008,notclickbait +Ex-Sheriff Guilty of Rape,notclickbait +"If You're Excited For The New ""Star Wars"" Movie You Have To See This Art",clickbait +This Oreo Popcorn Is The Movie Snack You Need Right Now,clickbait +"After Three Straight Titles, Jimmie Johnson Is Still Not Favored",notclickbait +"21 Real, Frustrating, And Neverending Struggles Of Being A Tidy Student",clickbait +US TV host Conan O'Brien leads in late night ratings,notclickbait +Artists Made These Heartbreaking Posters As Tribute To Child Victims Of The Peshawar School Attack,clickbait +14 Songs From The Early 2000s That Are Now Modern Day Christmas Classics,clickbait +17 Things People Who Don't Need Glasses Will Never Understand,clickbait +Deposed Mauritanian president Sidi Ould Cheikh Abdallahi is released,notclickbait +19 Truly Awkward Lube Confessions,clickbait +Rural African school goes to Sweden for environment award,notclickbait +"Boat in DR Congo capsizes, 80 feared dead",notclickbait +Research focuses on orchids mimicking female wasps,notclickbait +British Airways plans to cut carbon dioxide emissions by 50% by 2050,notclickbait +Congress Passes Bill to Tighten Weapons Programs,notclickbait +15 Boozy Cake Pops To Up Your Dessert Game,clickbait +"18 Gorgeous Santas That'll Make You Wanna Say ""Ho, Ho, Ho""",clickbait +Usain Bolt sets new world record in 100m sprint,notclickbait +Joachim to attempt breaking record for static cycling,notclickbait +The Secret To Delicious Cauliflower Pizza Is Lots And Lots Of Cheese,clickbait +21 Secrets A Third Wheel Will Never Tell You,clickbait +Maoists in India Briefly Seize Passenger Train,notclickbait +Chinese City Is Chilly to a Sex Theme Park,notclickbait +The Silence of Hybrids Causes Some Alarm,notclickbait +55 people die in Casablanca mattress factory fire,notclickbait +Coosh Headphones Are Meant to Stay Put,notclickbait +Freed Detainee Arrives in Britain,notclickbait +"39 Of The Most Heartfelt, Relatable And Comforting Quotes And Lyrics About Depression",clickbait +UK House of Commons' Speaker resigns,notclickbait +What Your Co-Workers On Facebook Need To Hear,clickbait +High-Tech Device Thumps Course to Test Its Firmness,notclickbait +This Is What Happens When Your Abuela Tries To Speak English,clickbait +Kendall Jenner And Gigi Hadid Transformed This Homeless Kid Into A Hot Model,clickbait +17 Photos That'll Remind You That The World Isn't All Bad,clickbait +"For Tiger Woods, Timing Is Everything",notclickbait +The Horrible Truth About Bagels,clickbait +Controversy after leak of preliminary report into Spanair disaster,notclickbait +Dulko Sends Sharapova to Another Early Exit,notclickbait +"How Well Do You Remember Season 5 Of ""The Walking Dead""",clickbait +These DIY Magnetic Jar Holders Will Save You So Much Cabinet Space,clickbait +"45 Things You Probably Didn't Know About ""The Sisterhood Of The Traveling Pants""",clickbait +Robber holds retired NYC police officer at gunpoint during convention,notclickbait +Keeping True Identity Online Becomes Battle,notclickbait +Tennessee Lieutenant Governor suggests that Islam is a 'cult',notclickbait +Relax And Watch This Hamster Eat A Little Carrot,clickbait +Bigoted woman: controversial Gordon Brown remarks caught on air,notclickbait +Apple swamped by iPhone 4 pre-orders,notclickbait +NASCAR: Edwards wins 2010 Kobalt Tools 500,notclickbait +Egyptian military appoints committee to amend constitution,notclickbait +"18 Things ""Back To The Future"" Forgot To Predict",clickbait +AKP secures mandate in Turkish general election,notclickbait +What Are Some Misconceptions About OCD,clickbait +19 Tips To Make Your Bed Even More Cozy,clickbait +Target Customers Were Confused When Porn Played Loudly Over The Intercom,clickbait +"Here Is ABC Family's ""Hocus Pocus"" Schedule For Halloween",clickbait +Live 8 concert plans announced,notclickbait +Canada pursues new nuclear research reactor to produce medical isotopes,notclickbait +"How Well Do You Remember The First Episode Of ""Friday Night Lights""",clickbait +Cher Destroys Donald Trump In One Little Tweet,clickbait +Nine Killed as Tornado Rakes Oklahoma,notclickbait +22 Brutally Honest Confessions From A Veterinarian,clickbait +These Valentine's Day Stories Prove That It's Maybe The Worst Day Of The Year,clickbait +New Zealand pilots receive bravery awards for foiling airliner hijack,notclickbait +Times Square bomb suspects arrested in Pakistan,notclickbait +19 People Who Contoured Better Than You In 2015,clickbait +"President Bush of the United States authorized NSA surveillance of citizens, bypassing court warrants",notclickbait +Fifteen flu sufferers die in Wales in one week,notclickbait +GLAAD Media Awards nominees announced,notclickbait +Senior citizen group seeks recovery of Medicare expenses from U.S. cigarette makers,notclickbait +Dion leads Ignatieff heading into final ballot of Canadian Liberal vote,notclickbait +France considers chemically castrating sex offenders,notclickbait +BBC and Sky networks reject Gaza appeal,notclickbait +Welsh soap opera Pobol y Cwm celebrates 35 years on air,notclickbait +21 Pictures Literally Every Girl Has Taken,clickbait +Airline bombing suspect spent months in Yemen,notclickbait +Here's Where You Should Be Shopping If You Love Victoria's Secret,clickbait +U.S. Diplomat Makes a Trip to Myanmar,notclickbait +"What Does It Mean To ""Hook Up"" With Someone",clickbait +Which Famous Movie Sidekick Are You Based On Your Zodiac,clickbait +Wikinews interviews Rich Mann and Kevin Smith of the United States Australian Football League about the upcoming National Championship,notclickbait +10 Trends That Didn't Catch On In 2015,clickbait +Solar powered plane completes first leg of transcontinental trip,notclickbait +"UK home shopping retailer Shop Direct group to cut 1,150 jobs",notclickbait +"This Sassy Line By Nadiya In This Week's ""Bake Off"" Is Splendid",clickbait +"Are You More Harry Or Marv From ""Home Alone?""",clickbait +Hong Kong 'tutor king' applies for bankruptcy,notclickbait +19 Passive Aggressive Notes That Deserve A Damn Medal,clickbait +"Before Kentucky Derby History Along the Rail, a Partnership Began in a Bar",notclickbait +Taliban Truce Seems in Flux in Pakistan,notclickbait +U.S. Senate approves revised bailout package after controversial additions,notclickbait +South African Prosecutors May Drop Fraud Charges Against A.N.C. Leader Jacob Zuma,notclickbait +A Disney Cover Band Got People To Kiss In New York City By Singing,clickbait +15 Instrumental Covers Of Pop Songs That Are Better Than The Original,clickbait +"On Hot Day, a Dash of Spice at Palmer Invitational",notclickbait +Google Maps incorporates satellite images,notclickbait +180 Hidden Miles of Great Wall of China Found,notclickbait +17 Ways To Be The Email Sender You Want To See In The World,clickbait +Romanian investor kidnapped in the Democratic Republic of the Congo is safe,notclickbait +"Here's How To Play ""Fallout 4"" As Imperator Furiosa",clickbait +Flooding Reshapes Waterfalls in Grand Canyon,notclickbait +9 Facebook Events You'd Actually Want To Attend,clickbait +Japanese tourist travels through 37 countries on just $2,notclickbait +Amy Poehler's Biggest Fear Is Making J-Lo Mad,clickbait +A Combative Trial in Colorado as a Controversial Ex-Professor Seeks to Win Back His Job,notclickbait +This Is What Someone With Anxiety Actually Hears,clickbait +Obama Seeks Vastly Expanded Afghan Force to Help Stabilize the Nation,notclickbait +21 Crazy Things People Have Actually Witnessed At Disney,clickbait +More Empty Shop Windows in New York,notclickbait +GigaPan Automates the Panoramic Picture,notclickbait +We Know Your Personality And Age Based On Your Favorite TV Show,clickbait +16 Things Dermatologists Want You To Know About Your Skin,clickbait +A 17-Year-Old Built A Bridge So That Underprivileged Kids Could Get To School Safely,clickbait +Euro 2008 Qualification: Finland defeats Kazakhstan 2-1 in Euro 2008 qualifier,notclickbait +What's The Best Halloween Costume Your Kid Has Ever Worn,clickbait +Obama Again Raises Estimate of Jobs His Stimulus Plan Will Create or Save,notclickbait +Watch These Grown-Ups Freak Out Over Virtual Puppies,clickbait +American pitchman Billy Mays dies at age 50,notclickbait +US political columnist James J. Kilpatrick dies at age 89,notclickbait +Officials report 29 rebels killed in Yemen after clashes,notclickbait +First Active offer 100% mortgages in Ireland,notclickbait +21 Absolutely Fucking Perfect Moments In Sporting Caption History,clickbait +Apple Inc. doubled its profits,notclickbait +Ukraine hurt by Russian gas deal,notclickbait +"13 Things That Will Definitely Maybe Happen In ""Game Of Thrones"" Season 6",clickbait +Can You Guess The Artist Based On Their Hit Album Cover Art,clickbait +These Google Calendar Tricks Will Get You Organized,clickbait +Near-Complete Fossil Offers Insight on Early Fish,notclickbait +Are These Pills Grey Or Red And Blue,clickbait +Al-Qaeda says another bin Laden tape to be released,notclickbait +Howard's 10 year party gatecrashed,notclickbait +Rep. Waxman: Karl Rove violated non-disclosure agreement,notclickbait +Former British Prime Minister Margaret Thatcher dies aged 87,notclickbait +23 Bizarre Beauty Trends From Yesteryear,clickbait +Yahoo May Announce Layoffs on Tuesday,notclickbait +Killen convicted of manslaughter for 1964 civil rights workers murders,notclickbait +A Shortcut To Hogwarts Has Been Spotted In Montreal,clickbait +21 Videos Of Eggs That Are Actually Porn,clickbait +Obama Embarks on His Honeymoon. The Question: How Long Will It Last?,notclickbait +11 Horrifyingly Cute Halloween Emojis,clickbait +This New Movie May Be The Best Use Of 3D Ever,clickbait +Honduran President Is Ousted in Coup,notclickbait +Tell Us About Yourself(ie): David Anders,clickbait +Rocket carrying NASA carbon dioxide satellite crashes into ocean,notclickbait +U.S. Senate defeats bill banning gay marriage,notclickbait +Which Famous Teenager Are You,clickbait +What Should Your Squad Be For Halloween This Year,clickbait +Jack Black Tried Halloween Pinterest Hacks And Became A Master Pinner,clickbait +"Male Mice Sing to Woo, but the Females Answer Just a First Call",notclickbait +"Marshall Eriksen's Best 25 Quotes On ""How I Met Your Mother""",clickbait +A Dying Mom Penned A Touching And Funny Note To Say Goodbye To Her Loved Ones,clickbait +21 Things You Should Actually Do If You Win The Lottery,clickbait +"Which Character From ""The Good Dinosaur"" Are You",clickbait +21 Life-Changing Health And Beauty Products You Should Try In 2016,clickbait +Venture capital investment in Ireland soars by 35%,notclickbait +Mongolia's ruling party wins elections as rioting subsides,notclickbait +"It Took Us A Serious Minute To Figure Out Who The Famous Person On The Cover Of ""Elle"" Is",clickbait +13 Divine Ways To Cook With Pomegranates This Holiday Season,clickbait +32 Pictures That Perfectly Sum Up Your Elementary School Experience,clickbait +"18 Slightly Horrifying Airport Moments That Will Make You Say ""WHY?!""",clickbait +Dolphin dies during performance at theme park,notclickbait +Suicidal man pushed off Chinese bridge,notclickbait +"Natalie Dormer Auditioned For A Completely Different Role On ""Game Of Thrones""",clickbait +What Breakup Song Would Taylor Swift Write About You,clickbait +"Ogilvy Lets Lead Dwindle, but Finishes on Top",notclickbait +"The ""X-Files"" Cast On When Mulder Fell In Love With Scully",clickbait +"Henry Cavill Is So Hot, He Can Make You Think Turtlenecks Are Attractive",clickbait +Keeping the News Crawl Running During Ad Breaks,notclickbait +19 Insanely Cute Trays That Will Help Keep Your Things Organized,clickbait +Two killed as Islamic militants storm Philippine jail,notclickbait +"In Debate on Spending, a Detour for Politics",notclickbait +"Identities of Bodies Found on Mesa Emerge, but Killer Remains Cloaked",notclickbait +Men Review Women's Sex Toys And Get Very Uncomfortable,clickbait +"Cleaning Cairo, but Taking a Livelihood",notclickbait +Watch 80 Years Of Halloween Costumes In Under Three Minutes,clickbait +Radioactive leakage at Swedish nuclear waste store,notclickbait +17 Things People Don't Understand About OCD,clickbait +Latest Citigroup Rescue May Not Be Its Last,notclickbait +Obama Reverses Rules on U.S. Abortion Aid,notclickbait +The New Wonder Woman Movie Officially Looks Amazing,clickbait +Disney Parodies The Biggest Movies Of 2015 In A Series Of Cute Film Posters,clickbait +13 Meet-Cute Stories That Actually Happened In Real Life,clickbait +65 Things That Make British People Feel Slightly Guilty,clickbait +"UK television presenter sacked after ""golliwog"" comment",notclickbait +"Steve Kubby, co-author of California Proposition 215, grows dangerously ill in US custody",notclickbait +"Reminder That Chad From ""High School Musical"" Is All Grown-Up",clickbait +"Urging Freedoms, Obama Chides Zimbabwe Leader",notclickbait +Mexican Data Show Migration to U.S. in Decline,notclickbait +68 pieces of luggage found behind Texas pet store,notclickbait +Many US TV stations preparing to make digital switch despite new legislation,notclickbait +Try Stuffing Chicken Parmesan Meatballs With Mozzarella And See What Happens,clickbait +13 Pedro Infante Songs To Listen To When You're Drunk AF,clickbait +"14 Things You Never Noticed During The Opening Number From ""Beauty And The Beast""",clickbait +"With Selection of Nicks, Giants Move On From Burress",notclickbait +12 Confessions From People Who Struggle With Anxiety,clickbait +2007 FIFA U17 World Cup: Germany eliminates Americans; sets up England quarter final,notclickbait +Justice Dept. Backs Saudi Royal Family on 9/11 Lawsuit,notclickbait +"Suspect in eight murders arrested in Illinois, United States",notclickbait +Billy Crystal: Feeling Lonely Behind the Facade,notclickbait +25 Things People With Thick Hair Can Simply Never Do,clickbait +I Won The Battle Against Button-Down Shirt Gapping For Under 15 Dollars,clickbait +Gwanda Chakuamba freed by Malawi court,notclickbait +Foul Ball Leaves Wright Limping; Hobbled Youkilis Is Out,notclickbait +Which Fantastic Magical Beast Are You Based On Your Zodiac Sign,clickbait +Attorney General Challenges Anti-Bias Law in California,notclickbait +Daring People Try Daring Shots,clickbait +Literally Just 22 Really Funny Tumblr Posts,clickbait +17 Heartbreaking Confessions From People Who Couldn't Love Someone Back,clickbait +9 Confessions That Will Make You Glad You're Single,clickbait +20 Tweets That Capture The Internet's Reaction To Salman Khan Being Acquitted,clickbait +"Do You Remember The Words To ""You're The One That I Want""",clickbait +Mongolia's ex-communists ahead going into Sunday's election,notclickbait +Asexual People Answer Questions You Have,clickbait +Yankees Hire Top Real Estate Brokerage to Market Its Premium Tickets,notclickbait +See 100 Years Of Japanese Beauty In Just Over One Minute,clickbait +Which Type Of Swearer Are You,clickbait +Two arrests made over Croxteth shooting,notclickbait +Taliban in Southern Pakistan Raise U.S. Fears,notclickbait +Concern about sovereign debt of some EU members roils markets,notclickbait +Are You More James Bond Or Austin Powers,clickbait +Up to 31 reported dead as Libyan government troops attack Misrata,notclickbait +11 Charts And Graphs Every James Bond Fan Will Get,clickbait +"Anushka Sharma Candidly, Thoroughly, PERFECTLY Called Out Bollywood's Sexism",clickbait +22 Reasons Life Was Easier In 2005,clickbait +Tell Us About A Time You Really Fucked Up,clickbait +This 29-Year-Old's Graphic Surgery Photos Show The Crippling Effects of Arthritis,clickbait +"Which Hogwarts Houses Do These ""American Horror Story"" Characters Belong In",clickbait +What Percent Single Are You,clickbait +Cyclone Kills Nearly 200 in Bangladesh and India,notclickbait +Russia assumes leadership of G8 for 2006,notclickbait +O. J. Simpson sentenced to 15 years in prison,notclickbait +"18 Times ""House M.D."" Got Way, Way Too Real",clickbait +23 Words That Mean Something Completely Different When You Get Engaged,clickbait +How Much Do You Actually Know About Ancient Technology,clickbait +These Pit Bull Brothers Are The Cutest Thing You'll See All Day,clickbait +The Try Guys Learn And Perform Iconic K-Pop Dances,clickbait +Nobel Peace Prize awarded to Kenyan environmental activist,notclickbait +Fifteen dead in Mexican car wash shooting,notclickbait +Australian States to launch high court battle against IR reforms,notclickbait +14 People Describe The Exact Moment They Realised They're In Love,clickbait +"Here's How To Survive Adulthood, According To Judah Friedlander",clickbait +"21 Reasons ""It's A Wonderful Life"" Is The Best Christmas Movie Of All Time",clickbait +Owner of Segway Jimi Heselden dies after scooter cliff fall aged 62,notclickbait +Protesters clash with police as Bush visits London,notclickbait +Windows 7 gets 'early release' in China; software pirates beat Microsoft to the punch,notclickbait +"9 Photos Of Successful Career Women, According To The Media",clickbait +Are These Smiths Lyrics Real Or Fake,clickbait +Supreme Is Tight But Maybe Not These Hoodies With Hentai On Them,clickbait +Report: invasion of Iraq provided boost for Al-Qaeda,notclickbait +Miss America Apologized To Vanessa Williams After 32 Years,clickbait +S.E.C. Nominee Offers Plan for Tighter Regulation,notclickbait +Medtronic Posts 69% Decline in Profit,notclickbait +This Mumbai Snapchatter Is Turning People's Selfies Into Super Cool Drawings,clickbait +Watch This Seahorse Shoot Baby Seahorses Out Of Its Stomach,clickbait +Afghanistan suicide bomb leaves seven Americans dead,notclickbait +22 Secrets Hotel Breakfast Waiters Are Supposed To Keep,clickbait +"Belgian king swears in PM Van Rompuy, cabinet",notclickbait +Met Victory With a Switch: The Hitters Bail Out Santana,notclickbait +Iran to Begin Tests at Nuclear Station,notclickbait +"Defending World Skating Title, Germans Take Pairs Lead",notclickbait +"State of emergency declared in Kingston, Jamaica",notclickbait +Which Iconic '00s Teen Movie Do You Actually Belong In,clickbait +Canada commits C$127.4 million to fighting tuberculosis,notclickbait +POLL: Which Female Celebrity Owned 2015,clickbait +Explosion in Russia kills 10: authorities suspect Chechen involvement,notclickbait +"All The ""Star Wars"" Fan Art You Didn't Know You Needed",clickbait +Which Golden Globe Red Carpet Look Are You Based On Your Zodiac Sign,clickbait +7 Surprisingly Easy Ways To Get Organized For The Holidays,clickbait +Path to the Supreme Court: Speak Capably but Say Little,notclickbait +"Bernie Sanders Played His Own Ancestor Alongside Larry David On ""SNL""",clickbait +Former Chilean President Pinochet suffers heart attack,notclickbait +Ad Losses Put Squeeze on TV News,notclickbait +Fats Domino missing in the wake of Hurricane Katrina,notclickbait +Meet The Inspiring Plus-Size Pole Dancer,clickbait +"17 Incredible ""Degrassi"" Fashion Moments",clickbait +"How To Be A Broadway Diva, Or At Least How To Fake It",clickbait +France beat Ireland in RBS Six Nations,notclickbait +The Hardest NHL Logo Quiz You'll Ever Take,clickbait +Homage to Chilean politician Jaime Guzmán nineteen years after his death,notclickbait +Mets Extend Winning Streak to Seven,notclickbait +An Inexpensive Digital Edge on the Golf Course,notclickbait +Teaching Economics and Pizza Equations,notclickbait +"Bush speaks of goals for U.S. withdrawal from Iraq, decries calls for timetable",notclickbait +New York City Mayor Michael Bloomberg spent US$108 million on third term bid,notclickbait +Is This Cover Art From A Kanye West Or Coldplay Album,clickbait +It's Time To Talk About How The Recast Dumbledore Was Literally The Worst,clickbait +Wikileaks.org restored as injunction is lifted,notclickbait +This Guy Proposed To His Girlfriend While On A Roller Coaster And Caught The Whole Thing On Camera,clickbait +"Michael J Fox And Christopher Lloyd Crashed ""Jimmy Kimmel Live"" As Marty McFly And Doc Brown",clickbait +Scottish Court Hears Appeal in 1988 Blast on Jetliner,notclickbait +"As Economy Slows, Americans Are Moving Less",notclickbait +10 Jeans Hacks To Keep Your Look On Point,clickbait +15 Life-Changing Stuffings Guaranteed To Impress Your In-Laws,clickbait +Scientists recreating the 1918 flu virus say 'it came from birds',notclickbait +UK Parliamentary Commissioner for Standards to investigate Nadine Dorries reality TV appearance,notclickbait +Ex-Lehman Trader Guilty of Fraud Charges,notclickbait +24 Things You Can Sympathise With If You've Ever Been Broke In London,clickbait +"24 Times The ""How To Get Away With Murder"" Cast Gave Us Serious Squad Goals",clickbait +"After Season in Flux, Knicks See Stability in Their Future",notclickbait +Golfer Tiger Woods delivers apology speech,notclickbait +The One Thing Guys Don't Talk About,clickbait +22 Home Decor Pieces That Will Make You Wish You Were Dead,clickbait +Computer Science Programs Make a Comeback in Enrollment,notclickbait +"Even Kim Kardashian Wishes The Kanye ""Rolling Stone"" Cover Was Real",clickbait +Senior Aide to Hussein Sentenced to 15 Years,notclickbait +Ted Kennedy diagnosed with brain tumor,notclickbait +Greece's Orthodox Church leader Christodoulos dies at age 69,notclickbait +You Definitely Need To Be Following January Jones On Instagram,clickbait +Adele Said A Mic Dropped In The Piano And That's Why Her Grammy Performance Was Off,clickbait +17 Colorado Trails That Should Be On Every Hiker's Bucket List,clickbait +Knife Attack Kills 3 at Belgian Day Care Center,notclickbait +17 Cats Really Loving The Cat Nip,clickbait +22 Hair Color Ideas For When You Can't Pick Just One,clickbait +Embassies across London lower their flags in honour of Nelson Mandela,notclickbait +How Well Do You Remember These PlayStation Games,clickbait +Creationist sentiments affect Imax business strategy,notclickbait +Ex-Chief of R.B.S. Agrees to Pension Cut,notclickbait +Bomb destroys Spanish police barracks and kills policeman in Basque Country,notclickbait +Here's Justin Bieber Looking Gorgeous While Skateboarding And Playing Instruments,clickbait +'Very serious': Chinese government releases corruption report,notclickbait +California judge disqualified from predatory lending case,notclickbait +Can You Name The John Green Book By Just The Cover Art,clickbait +Civilians dead following U.S. air strike on 'terrorist compound',notclickbait +Find Out Your Badass Harry Potter Roller Derby Name,clickbait +Maoist guerrilla attacks kill at least 17 on first day of Indian elections,notclickbait +"Facebook down for ""upgrades""; multiple blogs suggest site was hacked",notclickbait +18 Fall Salads You Need In Your Life Right Now,clickbait +25 Times Blue Ivy Had Her Shit Together Better Than You,clickbait +European airline Ryanair fined over ash-triggered flight cancellations,notclickbait +What's the Secret to Brawn's Success?,notclickbait +How A Gay Porn Star And Rapper Became Vine Famous And Then Liveblogged His Arrest On Facebook,clickbait +What Life-Changing Product Do You Wish You'd Owned Earlier,clickbait +17 Ridiculous Texts All Siblings Have Sent To Each Other,clickbait +An Adorable Animals Advent Calendar: December 21,clickbait +"18 Confessions That Prove You're Not Alone In Your ""The Bachelor"" Obsession",clickbait +A Streaky Phil Mickelson,notclickbait +VW and Porsche Merger Is Back on Track,notclickbait +"Sunni leader claims Iraq vote was a ""farce""",notclickbait +Coors Light Uses Cold to Turn Up Heat on Rivals,notclickbait +Taraji P. Henson Embraced Her Natural Beauty And Went Weave-Less For This Fashion Magazine,clickbait +The 12 Days Of Khaled,clickbait +This Is The Worst Version Of Your Favourite Mariah Carey Song,clickbait +"If Ryan Reynolds Could Give You Advice, What Would You Ask Him",clickbait +Vatican releases statement on health of Pope,notclickbait +Do You Have The Most Horrifying Cable Company Customer Service Story,clickbait +Instagram Star Reveals The Truth Behind Her Photos,clickbait +Zenit rocket explodes on launch pad,notclickbait +Lewis Hamilton wins 2008 Monaco Grand Prix,notclickbait +Prisons in New Zealand to introduce mobile phone jamming technology,notclickbait +What's SO Great About YOUR Life,clickbait +"I'm Not Antisocial, I Just Want To Be Alone",clickbait +Detroit Faces Its Critics With Anger and Tears,notclickbait +This Woman Uses Makeup To Make A Chilling Statement About Gender Stereotypes,clickbait +22 Medieval Satans Who Are Just Having A Really Bad Day,clickbait +Experts raise serious questions over safety of U.S. oil industry and warn another spill may be 'unavoidable',notclickbait +Thai court strips ex-Prime Minister of $1.4 billion,notclickbait +More Business Travelers Putting Away the Suitcase,notclickbait +Chinese human rights activist faces trial,notclickbait +Professor Accused of Pocketing NASA Money,notclickbait +Which Hogwarts House Do You Belong In Based On The First Letter Of Your Name,clickbait +24 Hilariously True Tweets For Retail Workers About The Plastic Bag Charge,clickbait +Opposition claims Australian Treasurer misled parliament,notclickbait +"Spend $10,600 on the Yankees, or on College or a Car?",notclickbait +Senator Says He Had Affair With an Aide,notclickbait +Australian Marcos Ambrose Is Rising Nascar Star,notclickbait +"Despite Odds, Cities Race to Bet on Biotech",notclickbait +"Highway bridge in Minneapolis, Minnesota, collapses",notclickbait +"Italy Upsets Canada, 6-2, at World Baseball Classic",notclickbait +Much of UK placed on flood alert by Met Office,notclickbait +Ousted Honduran president says crisis deal has failed,notclickbait +Only True New Jerseyans Will Ace This Quiz,clickbait +Can You Match The Song To The All Time Low Album,clickbait +21 Gifts For People Who Are Completely Obsessed With Tea,clickbait +Are Your Hook Up Habits Like Everyone Else's,clickbait +Two people confirmed dead in Boston Marathon bombing,notclickbait +27 Things You Need To Up Your Shower Game,clickbait +20 Miranda Sings Tweets To Brighten Your Monday,clickbait +China launches major cleanup operation after oil spill,notclickbait +Never Forget Hilary Duff Once Sucked A Dude's Thumb In A Music Video,clickbait +"Gordon Brown: Extra troops for Afghanistan, Yemen security the focus for int'l conference",notclickbait +Latvian government takes majority stake in Parex Bank,notclickbait +21 Unconventional Septum Rings That You Need Immediately,clickbait +Seven Canadians killed by Israeli air strike,notclickbait +U.S. jury deliberates immigrant smuggler case,notclickbait +Here's What You Need To Know About This Breathtaking Waterfall Castle,clickbait +Millions Of People Are In Love With This Elderly Man Waiting At The Airport,clickbait +We Can Determine If You're A Good Driver Or Not In Three Questions,clickbait +OMG The Middle Finger Emoji Has Arrived And Here's How To Get It,clickbait +Shares worldwide surge due to US government plan,notclickbait +Two N.H.L. Youngsters Are Giving Islanders a Reason to Hope,notclickbait +Controversial Florida attorney Jack Thompson disbarred,notclickbait +24 Times Nathan Fillion Was The Funniest Thing On Twitter,clickbait +Tee Times May Keep Some Players Under Rain Clouds,notclickbait +"Remember When David Bowie Was On ""SpongeBob Squarepants?""",clickbait +Fighting Around Afghanistan Leaves 50 Dead,notclickbait +"In Zimbabwe, Deal Likely to Fly or Fail This Week",notclickbait +UN High Commissioner for Human Rights: close Guantanamo Bay,notclickbait +35 Things That Will Hit You Right In The Childhood,clickbait +Types Of Co-Workers That Are The Absolute Worst,clickbait +Father throws 4-year-old daughter from bridge,notclickbait +A Black Woman Walks Into A Gun Show,clickbait +15 Times Calvin And Hobbes Reminded You To Never Stop Exploring,clickbait +Eurozone now officially in recession,notclickbait +Violence erupts in Guadeloupe after labor dispute,notclickbait +Three dead in murder-suicide shooting at Southern California fast food restaurant,notclickbait +Duke The Disabled Dwarf Bull Who Uses A Wheelchair To Walk Needs A New Home,clickbait +Joran van der Sloot charged with murder of Peruvian woman,notclickbait +"We Know Your Favorite ""Harry Potter"" Character Based On Your Favorite Puppy",clickbait +21 Signs You're Secretly A Real Bitch,clickbait +Multiple deaths as Congolese government cracks down on pro-democracy protests,notclickbait +Tens of thousands protest in London before Copenhagen climate change summit,notclickbait +Two Single AF People Got Married For A Week And Things Got Interesting,clickbait +Obama lessens US ban on offshore drilling,notclickbait +21 Words That Meant Something Completely Different In 2006,clickbait +When You Turn Into A Different Person While Driving,clickbait +"20 Things You've Always Wanted To Know About The 12th Doctor, As Told By Peter Capaldi",clickbait +"Things Got Rather Awkward When '90s Kids Danced To The ""Macarena"" Solo",clickbait +What Does Your Hair Say About You,clickbait +17 Times Titus Andromedon Lived Life To The Absolute Fullest,clickbait +Trustee May Take Helm of Union Local in Dispute,notclickbait +"With Surgery Done, Alex Rodriguez to Begin Rehab Immediately",notclickbait +The 45 Funniest Tweets From 2015,clickbait +"Here's What The ""Mona Lisa"" Looks Like With Celebrities' Brows",clickbait +Tell Us How You Started Your Big Weight Loss Journey,clickbait +"In Battle to Land Embattled Governor, Messages Show Extra-Sharp Media Elbows",notclickbait +Which 2009 Pop Hit Are You,clickbait +Canadian Conservatives vow to defend Arctic sovereignty,notclickbait +"Do You Actually Remember The First Episode Of ""The West Wing""",clickbait +13 Charts That Perfectly Sum Up Having A Big Butt,clickbait +Hurricane Wilma still a Category 5 threat,notclickbait +First Libertarian Party presidential nominee John Hospers dies at 93,notclickbait +8 Songs That Will Make You See Kesha In A New Light,clickbait +17 Things Only '00s Disney Kids Will Remember,clickbait +"Nearly 25,000 Iraqi civilians killed in Iraq, watchdog group claims",notclickbait +Thai PM sues newspaper for 500 million baht,notclickbait +Five beheaded during violent prison riot in Brazil,notclickbait +Google prepares to launch WiFi service,notclickbait +Pettitte Meets Prosecutors in Clemens Inquiry,notclickbait +Can You Get A Perfect Score On This California Quiz,clickbait +"Pinned Down, a Sprint to Escape Taliban Zone",notclickbait +Teamsters site suggests Tigger pardoned,notclickbait +Men Have Learned How To Braid Their Hair And They're Not Looking Back,clickbait +Australian Dollar reaches a 27 year high,notclickbait +Here's What It's Like When You Physically Can't Pierce Your Ears,clickbait +15 Hilarious Tweets About Narendra Modi Hanging Out With Francois Hollande,clickbait +Savings Plan: 5 Smart Steps,notclickbait +Lesbians Watched And Critiqued Lesbian Porn And It Was Traumatic,clickbait +"This Artist Painted Lin-Manuel Miranda's ""Hamilton"" On A Ten-Dollar Bill",clickbait +Justin Bieber Just Broke A Record Set By The Beatles,clickbait +What Does Your Choice Of Whiskey Say About You,clickbait +U.S. force-feeding Guantanamo hunger strikers,notclickbait +21 Times Neville Longbottom Out-Longbottomed Himself In 2015,clickbait +Who's the richest entertainer in Australia? The Wiggles of course!,notclickbait +This Band Surprised Dancers Auditioning For Their Music Video In The Best Way,clickbait +"U.S. watchdog group lists ""most corrupt members of Congress""",notclickbait +Your Chance to Show What the Super Bowl Logo Should Look Like,notclickbait +Which X-Men Character Are You Based On Your Zodiac Sign,clickbait +When You're A Girl With A Dirty Mind,clickbait +Find Your Next Great Book With The BuzzFeed Books Newsletter,clickbait +Hair-Raising Photos From The World Beard Championship 2015,clickbait +Finland win first ever Eurovision Dance Contest 2007 held in London,notclickbait +U.S. fighter jet crashes in Libya,notclickbait +We Know What Movie Makes You Cry,clickbait +Democrats Cut Labor Provision Unions Sought,notclickbait +"The Rise, Fall, And Improbable Comeback Of Morris Brown College",clickbait +Manitoba residents receive evacuation flood alerts,notclickbait +What Does Your Taste In Candy Say About Your Taste In Men,clickbait +29 Pictures That Will Finally Give You Some Goddamn Satisfaction,clickbait +Duke Tops Texas to Advance to Round of 16,notclickbait +"Human trafficking trial starts in Clearwater, Florida",notclickbait +Boorishness Works for Berlusconi,notclickbait +19 Painful Photos Every Short Guy Can Relate To,clickbait +34 Very Important Pictures Of Shia LaBeouf Watching His Own Movies,clickbait +Tour de France: Alberto Contador wins stage 14,notclickbait +Here's How To Do The Rainbow Snapchat Filter IRL For Halloween,clickbait +"Hospital in Essex, England fined £50,000 after patient dies due to health and safety breaches",notclickbait +Which Taylor Swift Tattoo Should You Get,clickbait +23 First Tweets From Famous Authors,clickbait +"Two people killed in aircraft crash in Hampshire, England",notclickbait +"Which ""Breaking Bad"" Character Are You Based On Your Favourite Colour",clickbait +13 Reasons Urinals Are The Worst Human Invention Of All Time,clickbait +"Ad Revenue Down, CBS Posts Profit Drop of 52%",notclickbait +New Yankee Stadium Has Room to Breathe and Room to Spend,notclickbait +24 Hilarious Tweets About Kids That Are Way Too Real For Every Parent,clickbait +"Adam Driver Is Bringing Emo Kylo Ren Vibes To ""SNL""",clickbait +"The Economy Is Bad, but 1982 Was Worse",notclickbait +The 19 Best Nonfiction Books Of 2015,clickbait +Teräsbetoni frontman J. Ahola on representing Finland at Eurovision 2008 & more,notclickbait +More Players Coming Out of Africa and Into M.L.S.,notclickbait +WHO director: Pandemic alert level will not be raised,notclickbait +How Much Do You Actually Know About Gay Sex,clickbait +The Definitive Guide To Kissing,clickbait +Orlando Ousts Defending Champions,notclickbait +8 Ways To Use Bi Invisibility To Your Advantage,clickbait +Cypriot court begins Greek air disaster trial,notclickbait +First Impressions Can Create Unrealistic Expectations for Young Basketball Recruits,notclickbait +These Pretzel Bites Are Dippable AF,clickbait +"In Gaza, the Wait to Rebuild Lingers",notclickbait +Champions League Final May Settle Ronaldo-Messi Debate,notclickbait +U.S. Scrambles for Information on Iran,notclickbait +"Parker and Duncan Play Well, but Spurs Are Pushed to Brink",notclickbait +"Can You Guess The ""Friends"" Season Based On Joey's Fridge",clickbait +Lift accident in Hong Kong skyscraper kills six,notclickbait +Man charged with murder of British woman in Hong Kong,notclickbait +US satellite radio provider Sirius unveils portable player,notclickbait +Iraqi Lawmaker Is Charged With Masterminding an Attack on Parliament,notclickbait +27 Times BBC News Failed So Hard It Just Failed,clickbait +Puerto Rico Ex-Governor Is Acquitted of Graft,notclickbait +What New Song Should Be Your Jam This Weekend,clickbait +Scientists improve cancer research techniques,notclickbait +"24 Reasons To Start Watching ""Vicious"" Right Now",clickbait +Bartali-Coppi Rivalry Still Stirs Passion at the Giro,notclickbait +"38 Questions From ""Friday Night Lights"" We Still Need An Answer To",clickbait +15 Of The Most Bengali Things That Happened In 2015,clickbait +You Won't Be Able To Stop Watching This Mesmerising Rangoli Video,clickbait +15 Invisible Things You Definitely Missed At The 2nd GOP Debate,clickbait +"Olivia From ""The Bachelor"" Made It Off That Beach They Left Her On",clickbait +Gaps Appear in G.O.P. Solidarity,notclickbait +Please Help Us Figure Out What These Carrie Fisher Tweets Mean,clickbait +Don't Try To Make Your Own Homemade Creme Eggs Because It's More Trouble Than It's Worth,clickbait +NBC Still Waiting for a Hit From Its Hit Maker,notclickbait +"The First ""Grease: Live"" Teaser Is Here And It's Absolutely Electrifying",clickbait +17 Mindblowing Ways To Eat Bagels,clickbait +13 Things All Slytherins Will Need In 2016,clickbait +"Chicago Metra considers selling naming rights for train lines, stations",notclickbait +Jets Hope N.F.L. Draft Helps Rebuild Offense,notclickbait +"British seaman appears in court, charged with murder after shooting on nuclear submarine",notclickbait +9 Things You'll Definitely Lose In 2016,clickbait +Elections held in Bulgaria,notclickbait +Fed Calls Gain in Household Wealth a Mirage,notclickbait +"For Poodles, Maintenance of the Highest Order",notclickbait +Can You Help People Find This Kid So They Can Give Him A Real Messi Shirt,clickbait +British Court Finds 3 Not Guilty of Aiding London Bombers,notclickbait +"Fuck, Marry, Kill: Same Actor, Different Characters",clickbait +"We Need To Talk About The Music From ""A Goofy Movie""",clickbait +U.S. Helps Palestinians Build Force for Security,notclickbait +35 Costumes Guaranteed To Win Halloween,clickbait +"Former Ukraine PM Yulia Tymoshenko to end hunger strike, daughter announces",notclickbait +We Invaded Three Queer Lady Couple's Apartments To See What It's Like After The U-Haul,clickbait +This Is What Happens When You're A Black Girl Who Can't Dance,clickbait +Nigerian sentenced to death for admitting to gay relations,notclickbait +"27 ""Scrubs"" Moments That Will Make You Laugh Every Time",clickbait +"Poker's all about luck, says Swiss Supreme Court",notclickbait +Basketball: Lakers score 102 to defeat the Celtics in Game 1 of the 2010 NBA Finals,notclickbait +Fiat Gives Detroit a Lesson on Small Cars,notclickbait +Angolan police arrest two after attack on Togo football team,notclickbait +Massachusetts study finds links between bullying and family violence,notclickbait +15 Hilariously Awesome Stoner Halloween Costume Ideas,clickbait +We Need To Talk About Mariah Carey,clickbait +Students Have A Legal Right To Safe Spaces,clickbait +Wikimania jury chooses Buenos Aires for 2009 location,notclickbait +Scholarships awarded to isolated pacific islanders in Micronesia,notclickbait +34 Non-American Musicians You Need To Add To Your Playlist,clickbait +Pakistan arrests 35 people suspected in US soldiers' death,notclickbait +Renault F1 launch criminal complaint against former driver over race-fixing allegation,notclickbait +How Much Do You Actually Know About Cooking,clickbait +Show Us The Best Halloween Costume You've Ever Made,clickbait +This Dog Loves Leaves More Than You Love Anything,clickbait +An Eight-Months Pregnant Bridesmaid Went Into Labor The Morning Of Her Friend's Wedding,clickbait +Markets dragged down by credit crisis,notclickbait +"If You Liked ""Uptown Funk,"" You'll Love This Song",clickbait +French Montana Wearing Slippers On The Beach Will Make You Feel Uncomfortable,clickbait +23 Things You'll Only Relate To If You're Slightly Obsessed With Makeup,clickbait +The Wait Is Over: You Can Now Put Your Penis In A Mouth-Vagina-Anus Robot,clickbait +11 Robots Who Will Certainly Rise Up To Destroy All Humans,clickbait +Victims of London jetliner crash sue Boeing,notclickbait +Japanese stocks continue to fall after earthquake,notclickbait +Six H1N1 cases appear in the Philippines,notclickbait +"Here's What ""Harry Potter"" Characters' Eyes Are Actually Supposed To Look Like",clickbait +9 Illustrated Tweets Guaranteed To Make You LOL,clickbait +New York Assembly passes same-sex marriage bill,notclickbait +9 Slightly Gross Reasons You Might Want To Leave Your Pubes The Fuck Alone,clickbait +Pakistani Prime Minister agrees to put all state executions on hold,notclickbait +Judge Tosses Key Evidence Against Bonds,notclickbait +8 WTF Moments Of The 2016 Presidential Race,clickbait +Russia Ends Operations in Chechnya,notclickbait +How Dad Is Your Dad,clickbait +RTÉ's Eddie Hobbs attracts massive audience,notclickbait +What Kind Of Pop-Tart Matches Your Personality,clickbait +"Scotland Yard says suicide bomb blast killed Bhutto, not bullet",notclickbait +We Know Whether You're An Extrovert Or Introvert Based On Your Instagram,clickbait +New York Times reporter rescued in Afghanistan,notclickbait +Serbian firefighters shot at from Kosovo,notclickbait +Zimbabwe opposition claims early victory in election,notclickbait +"First, Third or Outfield, Mets Want Tatis in a Position to Hit",notclickbait +"Is This Celeb Danish, Norwegian, Or Swedish",clickbait +17 Moments That Are Too Real For Anybody Who's Not Totally Out Of The Closet,clickbait +Believers Play The Ouija Board,clickbait +Here's What 19 Painfully Handsome Male Models Look Like Without Makeup,clickbait +8 Teams Threaten to Splinter From Formula One,notclickbait +Defense Chief Criticizes Bid to Add F-22s,notclickbait +A Manitoba Photographer Is Selling A Gorgeous Calendar To Help Refugees,clickbait +Regulator bans UK video-on-demand service,notclickbait +21 Badass Things Bollywood Did In 2015,clickbait +13 Weirdly Random Things Indians Have Uploaded To Porn Sites,clickbait +Abuse Photos Part of Agreement on Military Spending,notclickbait +"Though Many Are Stalked, Few Report It",notclickbait +Black Friday Is Your Chance To Get The Naked Palettes On Sale,clickbait +Here's Why You Actually Get Depressed In The Winter,clickbait +28 Memes That Pretty Much Sum Up Life In 2015,clickbait +"Men, Stephen King Has A Really Important Message For You",clickbait +19 Things That Happen When You Get Together With Your Uni Friends,clickbait +US Senate offices evacuated,notclickbait +"What Planet In The ""Star Wars"" Galaxy Is This",clickbait +16 Words That Mean Something Completely Different At Target,clickbait +Signs of Hope Emerge in the West Bank,notclickbait +The Popular Newsweekly Becomes a Lonely Category,notclickbait +Xbox 360 shortages expected on debut day in Europe,notclickbait +14 Comics That Capture Your Magical Gay Experience,clickbait +16 Jackie Chan Stunts That Perfectly Capture Your Twenties,clickbait +7 Bizarre Facts You Didn't Know About This Week In History,clickbait +"F1: Massa wins 2008 Brazilian Grand Prix, Hamilton wins championship",notclickbait +"Which ""Friends"" Character Should You Actually Date",clickbait +Can We Ask You A Really Weird Question,clickbait +26 Breathtaking U.S. Sights You Need To See Before You Die,clickbait +Controversial cancer test gains support,notclickbait +27 Portraits From Wes Anderson Movies That Are Beautifully Striking,clickbait +13 Things You Should Never Say To A Latina When Dating,clickbait +2008 Taipei AMPA: IT industry will pulse the growth of automotive industry,notclickbait +Iran building collapse kills 19,notclickbait +French Find Safety Nets Multiplying in Pastures,notclickbait +10 Places To Travel In 2016,clickbait +New findings suggest AIG executive bonuses were larger than previously thought,notclickbait +"Try To Keep Your Chill, But Colin Farrell's Hair Is Turning Daddy Gray",clickbait +North Korea prepared for 'sacred' nuclear war with South,notclickbait +Americans Try Unusual Japanese Food Inventions,clickbait +Iran tests new centrifuges,notclickbait +Dating Struggles For Extroverted Ladies,clickbait +14 People Share What It's Like To Be Divorced At A Young Age,clickbait +9 Women In Movies Who Somehow Brought In All The Money This Year,clickbait +South Africa Bars Dalai Lama From a Peace Conference,notclickbait +"Swine Flu Vaccine May Be Months Away, Experts Say",notclickbait +"Michael B. Jordan Got Laid The Fuck Out While Filming ""Creed""",clickbait +Coral Study in Mexico Suggests Rapid Sea Rise From Warming,notclickbait +This British Rapper Is Trying To Find Out Why His Father Killed Himself,clickbait +Barclays Says It Will Not Seek Bailout,notclickbait +What Brave Thing Should You Do Today,clickbait +27 Wonderfully Geeky Products You Never Knew Your Kitchen Needed,clickbait +"Two killed, two seriously injured after boulder collapses onto house in Stein an der Traun, Germany",notclickbait +Controversial Boxer Tyson Fury Stripped Of Heavyweight Title Just Two Weeks After Winning It,clickbait +These Are The 2016 Books Readers Are Most Pumped For,clickbait +"Which Song From ""Steven Universe"" Are You",clickbait +31 Childhood Photos Of Amy Schumer That Are Just Too Damn Cute,clickbait +"The First ""Finding Dory"" Trailer Is Here And It Suggests How Dory Went Missing",clickbait +A More Assertive Donovan Drives U.S. Soccer Ambitions,notclickbait +"Fatal train fire may have been suicide attempt, British police say",notclickbait +Which Fictional Queer Lady Are You Based On Your Zodiac Sign,clickbait +19 Ideas So Bad Only The Internet Could Love Them,clickbait +Watch This Out Gay Marine Ask Singer Steve Grand To The Marine Corps Ball,clickbait +Hualien warm-up of 2007 ING Taipei Marathon kicked off,notclickbait +Former NBA ref surrenders to charges he bet on games he officiated,notclickbait +Sri Lankan Troops Breach Rebel Haven,notclickbait +The King's Speech wins People's Choice Award at 2010 Toronto Film Festival,notclickbait +Liz Claiborne Looks for a Mizrahian Lift,notclickbait +Which TV Character Alter Ego Are You,clickbait +India and China to discuss energy: cooperation or competition?,notclickbait +Which Minor Minor Character From 30 Rock Are You,clickbait +Ukraine Says 3 Tried to Sell Bomb Material,notclickbait +Who Said It: The Disney Insult Edition,clickbait +What Does Your Taste In Dessert Say About You,clickbait +The Recession May Be a Boon to Book Sales,notclickbait +Human Error and Management Decisions Faulted in W. Va. Bayer Explosion,notclickbait +Australian rules football: Drouin through to 2010 Gippsland Football League Grand Final,notclickbait +LGBT Women Share Things They Are Tired Of Hearing In This Powerful Video,clickbait +This Is What My Mother's Eating Disorder Has Taught Me,clickbait +We Know Your Age Based On Your Taste In Men,clickbait +"22 Reasons Fans Are Obsessed With ""The Walking Dead""",clickbait +How Do You Stay Fit In College,clickbait +"Afghans, From Hospital Beds, Dispute U.S. Account of Raid",notclickbait +Meet One Of The Filthiest Female Characters You'll See Onscreen This Year,clickbait +"Which ""Friday Night Lights"" Quarterback Is Your Soulmate",clickbait +Here's A GIF That Perfectly Captures Joke-Stealing On Twitter,clickbait +Alassane Ouattara 'wins' Ivory Coast presidential election,notclickbait +Natalie Portman Looks Exactly Like Jackie Kennedy In Her New Biopic,clickbait +"Dannie Abse hurt, wife killed in car accident",notclickbait +Macy's flagship store in New York evacuated due to fire,notclickbait +People Try Snapchat's New Selfie Filters,clickbait +Long Line at the N.B.A. Complaint Desk,notclickbait +DNA components found in meteorites,notclickbait +International experts probe deadly Ebola Reston virus outbreak in Philippine pigs,notclickbait +Headless man found near suburban Chicago school,notclickbait +23 Beautiful Cakes That Will Make Your Dreams Come True,clickbait +Why Umbrellas Are Just The Worst,clickbait +"Glasgow, Scotland shooting leaves two hospitalised",notclickbait +Neil Patrick Harris And David Burtka Made Their Kids An Adorable Sandcastle Cake,clickbait +Australian Football League to hold match in China,notclickbait +Five Jet Crew Members Have Been Suspended For Allowing Sonu Nigam To Sing On A Flight,clickbait +U.S. Retail Sales Rose in May,notclickbait +17 Times The Internet Perfectly Summed Up Being A Cancer,clickbait +Somali opposition group al-Shabaab to block WFP food aid,notclickbait +Barack Obama accepts US presidential nomination from the Democratic Party,notclickbait +250 Feared Dead in Indonesia Ferry Disaster,notclickbait +"Bristol Harbour Festival 2008 lauched, 200 thousand expected to attend",notclickbait +19 Foods To Shut Yourself In With This Winter,clickbait +1 Trip To Trader Joe's + $20 = 5 Easy Vegan Dinners,clickbait +"Small explosion investigated in Times Square, New York",notclickbait +Sarkozy and Merkel Try to Shape European Unity,notclickbait +23 Gifts That Will Make All Makeup Lovers Swoon,clickbait +17 Faces That Perfectly Sum Up Every Black Girl's Transition From Relaxed To Natural Hair,clickbait +How 2015 Were You,clickbait +Nokia takes over Symbian OS development,notclickbait +North Korea Says Journalists Admitted Crimes,notclickbait +19 Cat Reactions For Every Thanksgiving Situation,clickbait +Kate Winslet Keeps Her Oscar In Her Bathroom So People Can Practice Their Own Acceptance Speeches,clickbait +9 Gross Things To Never Do In Front Of Your Man,clickbait +U.N. Leads Evacuation From Sri Lanka,notclickbait +"One Direction's New Music Video For ""Perfect"" Is Actually Perfection",clickbait +Japan earthquake shifts Earth's axis 10 centimetres,notclickbait +Somalia food hijackers make new demands,notclickbait +"How Well Do You Remember The ""Filthy Animal"" Scene From ""Home Alone""",clickbait +G8 finance ministers agree on deal to relieve debts of 18 poorest nations,notclickbait +Increased tension in border dispute between Eritrea and Ethiopia,notclickbait +18 Times James Blunt Took No Shit On Twitter In 2015,clickbait +South Ossetia says it will join North Ossetia-Alania as a federal subject of Russia,notclickbait +We Know Your Favorite Disney Dude Based On Your Zodiac Sign,clickbait +30 Things You Need To Cook In November,clickbait +Even Well-off Consumers Aim to be Less Conspicuous,notclickbait +Chris Moyles Is Returning To Radio With A Breakfast Show On Radio X,clickbait +What's Your Teen Name,clickbait +19 Things You Probably Never Knew About Nightmares,clickbait +17 Twentysomething Problems Summed Up By Houseplants,clickbait +How To Say No,clickbait +Ignored warnings 'worsened' situation in Myanmar,notclickbait +Lions Keep Up Their Losing Ways,notclickbait +UNESCO votes in favor of Palestine membership,notclickbait +12 Aussies Reveal What They Love And Hate About Britain,clickbait +"20,000 Californian state workers may lose their jobs",notclickbait +UK retailers MFI and Woolworths collapse,notclickbait +What Taylor Swift Lyric Should Be Your Life Motto,clickbait +Does Black Ever Crack? -- Guess These Black Celebs Ages,clickbait +Which Rihanna Album Are You Actually,clickbait +Do You Have The Best Photo Of A Kid Freaking Out With Santa,clickbait +Tell Us About Your Self(ie): Sabrina Carpenter,clickbait +"11 Internet Cartoons That Will Make You Feel ""Popeful""",clickbait +Pigskin A Blanket: NFL Week 2 Picks,clickbait +"Tanker crash kills two in Brevard County, Florida",notclickbait +17 Pictures Short Girls Will Never Understand,clickbait +New U.S. immigration bill proposes time-limit and employer scrutiny,notclickbait +Shelling in Somalia's capital kills twenty,notclickbait +John Stamos Blessed The World With A Photo Of His Bare Ass For Paper Magazine,clickbait +Malaysian Court Orders Reinstatement of Opposition Minister,notclickbait +"2015 You're Great And I'ma Let You Finish, But 2006 Had The Best Music Of All Time",clickbait +"After uncertain day of Eurovision rehearsals, EBU will place sanctions on Spain and RTVE",notclickbait +22 Faces That Perfectly Capture The Struggle Of Trying To Act Sober,clickbait +21 Things Only Brutally Honest Women Will Understand,clickbait +"$8,000 Phobos Media Center Comes With a Tech",notclickbait +Tinder Just Added An STI Testing Locator To Their Website,clickbait +12 Animals Who Can't Tell What You Are For Halloween,clickbait +7 Mental Health Resources For When You're Feeling Helpless,clickbait +Former US Vice President Dick Cheney: 'Barack Obama is a one-term President',notclickbait +Brazilian soccer player's mother freed by kidnappers,notclickbait +21 Cosy AF Bedroom Goals,clickbait +27 Things You'll Only Understand If You're Slightly Obsessed With Nail Polish,clickbait +25 Feminist Songs That Always Make You Feel Like A Boss,clickbait +13 Struggles Only Pets Who Live In Canada Will Understand,clickbait +Nigeria's cabinet dissolved by acting president,notclickbait +Obama gives check to Clinton campaign,notclickbait +At least thirty-three more dead in Ethiopia election clashes,notclickbait +Groundhogs Aren't The Only Animals Who Can Predict The Future,clickbait +Wikinews interviews Finnish 'Rock 'N' Troll' band Kivimetsän Druidi,notclickbait +Thousands protest privatisation of Australian electricity industry,notclickbait +Which Adele Lyric Describes Your Love Life,clickbait +"21 Places Chandler Bing Might Be Instead Of Reuniting With The ""Friends"" Cast",clickbait +"Massachusetts to set up ""village"" for Katrina evacs",notclickbait +Request Timeout,clickbait +How Many Of These Defunct English Football Grounds Have You Been To,clickbait +Dead body left in UK hospital alongside living patients for seven hours,notclickbait +South Africa defeats Australia in second cricket test,notclickbait +11 Types Of Student Everyone Meets At University,clickbait +19 Brutally Honest Confessions From A Tesco Worker,clickbait +Verizon Wireless VoIP Device Cuts the Landline,notclickbait +"UK government to spy on phone, email, browsing, of entire population",notclickbait +"PSA: Beware The ""Star Wars"" Hype",clickbait +"Boat in Bangladesh sinks, at least twelve dead",notclickbait +23 Annoying Things That Will Happen For The Rest Of Your Life,clickbait +23 Crucial Items Every Die-Hard Tailgater Needs,clickbait +"The ""I Think I Hear Your Girlfriend Calling You"" Puzzle",clickbait +15 Times Chris Traeger Inspired You To Be Your Best Self,clickbait +US Senate advances health care reform bill,notclickbait +These Inspiring Portraits Capture The Emotional Fight Against Cancer,clickbait +Student on Trial in Italy Claims Police Pressure,notclickbait +Felipe Massa wins 2008 Turkish Grand Prix,notclickbait +17 Books You Definitely Read If You Grew Up In The '00s,clickbait +Here's Justin Trudeau Walking The Wrong Way Up An Escalator For All Eternity,clickbait +"Amid Crackdown, Iran Admits Voting Errors",notclickbait +21 people killed and 113 reported injured in three blasts in Mumbai,notclickbait +"SOUND THE ALARMS: Pepsi Perfect From ""Back To The Future Part II"" Is Finally Being Released",clickbait +This Duo Is Travelling Across India For A Year To Highlight The Issues Nobody's Talking About,clickbait +16 Teens Who Are Going To Rule The World In 2016,clickbait +18 Of The Most Cringeworthy Lyrics From 2015,clickbait +Woman in Binghamton Athletics Files Harassment Complaint,notclickbait +"Which Moment From ""The Simpsons"" Always Makes You Ugly Cry",clickbait +Would You Be A Better Date Than My New Man,clickbait +16 Photos That Prove Just How Hardcore Curling Is,clickbait diff --git a/example/data/training_tasks.csv b/example/data/training_tasks.csv new file mode 100644 index 0000000..01fde84 --- /dev/null +++ b/example/data/training_tasks.csv @@ -0,0 +1,11 @@ +INPUT:headline,GOLDEN:category +Gunman on the loose in Melbourne,notclickbait +Extreme Phone Pinching Is Possibly The Most Nerve-Racking Internet Craze Yet,clickbait +Don't Try To Make Your Own Homemade Creme Eggs Because It's More Trouble Than It's Worth,clickbait +Hong Kong 'tutor king' applies for bankruptcy,notclickbait +Signs of Hope Emerge in the West Bank,notclickbait +UConn Men Overpower Texas A&M to Reach Round of 16,notclickbait +2015's Biggest Memes Are Now Emojis,clickbait +Amy Schumer's Press Room Interview After She Won Her Emmy Is Iconic,clickbait +12 Confessions From People Who Struggle With Anxiety,clickbait +Even Well-off Consumers Aim to be Less Conspicuous,notclickbait diff --git a/example/src/Dockerfile b/example/src/Dockerfile new file mode 100644 index 0000000..2fad0b3 --- /dev/null +++ b/example/src/Dockerfile @@ -0,0 +1,21 @@ +FROM ubuntu:20.04 + +RUN apt update +RUN apt install -y software-properties-common +RUN apt install -y python3.9 +RUN apt install -y python3-pip + +RUN pip3 install numpy pandas scikit-learn toloka-kit crowd-kit nltk +RUN python3 -c "import nltk; nltk.download('stopwords')" + + +WORKDIR /code + +ADD src/create_project.py /code/create_project.py +ADD src/create_pool.py /code/create_pool.py +ADD src/create_training.py /code/create_training.py +ADD src/create_tasks_from_csv.py /code/create_tasks_from_csv.py +ADD src/wait_pool.py /code/wait_pool.py +ADD src/aggregate_assignments.py /code/aggregate_assignments.py +ADD src/concatenate_datasets.py /code/concatenate_datasets.py +ADD src/train_test_model.py /code/train_test_model.py diff --git a/example/src/aggregate_assignments.py b/example/src/aggregate_assignments.py new file mode 100644 index 0000000..2ad109c --- /dev/null +++ b/example/src/aggregate_assignments.py @@ -0,0 +1,36 @@ +import argparse +import os.path + +import pandas as pd +from crowdkit.aggregation import DawidSkene +from toloka.client import TolokaClient + + +def aggregate_assignments(toloka_token: str, pool_id: str) -> pd.DataFrame: + toloka_client = TolokaClient(toloka_token, 'PRODUCTION') + assignments = toloka_client.get_assignments_df(pool_id) + assignments = assignments[assignments['GOLDEN:category'].isna()] # Ignore honeypots. + assignments = assignments.rename(columns={'INPUT:headline': 'task', + 'OUTPUT:category': 'label', + 'ASSIGNMENT:worker_id': 'performer'}) + df = DawidSkene(n_iter=20).fit_predict(assignments).to_frame().reset_index() + df.columns = ['headline', 'category'] + return df + + +if __name__ == '__main__': + # command line arguments + parser = argparse.ArgumentParser(description='Aggregate assignments from Pool on Toloka.') + parser.add_argument('--token_env', type=str, required=True, help='Environment variable for Toloka API access token') + parser.add_argument('--pool_id_path', type=str, required=True, help='Path to file with pool id') + parser.add_argument('--aggregated_tasks_path', type=str, required=True, help='Path to store aggregation results') + args = parser.parse_args() + + toloka_token = os.environ[args.token_env] + + with open(args.pool_id_path) as pool_id_file: + pool_id = pool_id_file.readline().strip() + + aggregation_results = aggregate_assignments(toloka_token, pool_id) + + aggregation_results.to_csv(args.aggregated_tasks_path, index=False) diff --git a/example/src/concatenate_datasets.py b/example/src/concatenate_datasets.py new file mode 100644 index 0000000..33ed23a --- /dev/null +++ b/example/src/concatenate_datasets.py @@ -0,0 +1,13 @@ +import argparse + +import pandas as pd + +if __name__ == '__main__': + # command line arguments + parser = argparse.ArgumentParser(description='Concatenate two datasets.') + parser.add_argument('--datasets', nargs='+', type=str, required=True, help='Paths to datasets') + parser.add_argument('--output', type=str, required=True, help='Path to output file') + args = parser.parse_args() + + datasets = [pd.read_csv(path) for path in args.datasets] + pd.concat(datasets).to_csv(args.output, index=False) diff --git a/example/src/create_pool.py b/example/src/create_pool.py new file mode 100644 index 0000000..1891dfd --- /dev/null +++ b/example/src/create_pool.py @@ -0,0 +1,40 @@ +import argparse +import os +from datetime import datetime, timedelta + +from toloka.client import TolokaClient, Pool + + +def create_pool(toloka_client: TolokaClient, pool_config_path: str, project_id: str, training_id: str) -> Pool: + with open(pool_config_path) as pool_config_file: + json_string = pool_config_file.read() + pool = Pool.from_json(json_string) + pool.project_id = project_id + pool.quality_control.training_requirement.training_pool_id = training_id + pool.will_expire = datetime.now() + timedelta(days=10) + return toloka_client.create_pool(pool) + + +if __name__ == '__main__': + # command line arguments + parser = argparse.ArgumentParser(description='Create a pool in Toloka project.') + parser.add_argument('--token_env', type=str, required=True, help='Environment variable for Toloka API access token') + parser.add_argument('--project_id_path', type=str, required=True, help='Path to file with project id') + parser.add_argument('--training_id_path', type=str, required=True, help='Path to file with training id') + parser.add_argument('--pool_config_path', type=str, required=True, help='Path to a pool config file') + parser.add_argument('--pool_id_path', type=str, required=True, help='Path to store created pool id') + args = parser.parse_args() + + toloka_token = os.environ[args.token_env] + + with open(args.project_id_path) as project_id_file: + project_id = project_id_file.readline().strip() + + with open(args.training_id_path) as training_id_file: + training_id = training_id_file.readline().strip() + + toloka_client = TolokaClient(toloka_token, 'PRODUCTION') + pool = create_pool(toloka_client, args.pool_config_path, project_id, training_id) + + with open(args.pool_id_path, 'w') as pool_id_file: + pool_id_file.write(pool.id) diff --git a/example/src/create_project.py b/example/src/create_project.py new file mode 100644 index 0000000..f20df12 --- /dev/null +++ b/example/src/create_project.py @@ -0,0 +1,28 @@ +import argparse +import os + +from toloka.client import TolokaClient, Project + + +def create_project(toloka_client: TolokaClient, project_config_path: str) -> Project: + with open(project_config_path) as project_config_file: + json_string = project_config_file.read() + project = Project.from_json(json_string) + return toloka_client.create_project(project) + + +if __name__ == '__main__': + # command line arguments + parser = argparse.ArgumentParser(description='Create project on Toloka.') + parser.add_argument('--token_env', type=str, required=True, help='Environment variable for Toloka API access token') + parser.add_argument('--project_config_path', type=str, required=True, help='Path to a project config file') + parser.add_argument('--project_id_path', type=str, required=True, help='Path to store created project id') + args = parser.parse_args() + + toloka_token = os.environ[args.token_env] + + toloka_client = TolokaClient(toloka_token, 'PRODUCTION') + project = create_project(toloka_client, args.project_config_path) + + with open(args.project_id_path, 'w') as project_id_file: + project_id_file.write(project.id) diff --git a/example/src/create_tasks_from_csv.py b/example/src/create_tasks_from_csv.py new file mode 100644 index 0000000..1373cce --- /dev/null +++ b/example/src/create_tasks_from_csv.py @@ -0,0 +1,123 @@ +import argparse +import os +from enum import Enum +from typing import List, Optional + +import pandas as pd +from toloka.client import TolokaClient, Task + + +class TaskType(Enum): + POOL = 'POOL' + TRAINING = 'TRAINING' + HONEYPOT = 'HONEYPOT' + + +def init_pool_tasks(tasks_df: pd.DataFrame, pool_id: str) -> List[Task]: + tasks: List[Task] = [] + + headings = [column_name for column_name in tasks_df.columns if column_name.startswith('INPUT:')] + + for _, row in tasks_df.iterrows(): + tasks.append( + Task(input_values={field[len('INPUT:'):]: row[field] for field in headings if not pd.isna(row[field])}, + pool_id=pool_id)) + + return tasks + + +def init_control_tasks(tasks_df: pd.DataFrame, pool_id: str) -> List[Task]: + tasks: List[Task] = [] + + input_headings = [column_name for column_name in tasks_df.columns if + column_name.startswith('INPUT:')] + golden_headings = [column_name for column_name in tasks_df.columns if + column_name.startswith('GOLDEN:')] + + for _, row in tasks_df.iterrows(): + known_solutions = [{'output_values': {field[len('GOLDEN:'):]: row[field] for field in golden_headings if + not pd.isna(row[field])}}] + tasks.append(Task( + input_values={field[len('INPUT:'):]: row[field] for field in input_headings if not pd.isna(row[field])}, + known_solutions=known_solutions, + pool_id=pool_id)) + + return tasks + + +def init_training_tasks(tasks_df: pd.DataFrame, pool_id: str) -> List[Task]: + tasks: List[Task] = [] + + input_headings = [column_name for column_name in tasks_df.columns if + column_name.startswith('INPUT:')] + golden_headings = [column_name for column_name in tasks_df.columns if + column_name.startswith('GOLDEN:')] + hint_headings = [column_name for column_name in tasks_df.columns if + column_name.startswith('HINT:')] + + for _, row in tasks_df.iterrows(): + known_solutions = [{'output_values': {field[len('GOLDEN:'):]: row[field] for field in golden_headings if + not pd.isna(row[field])}}] + if len(hint_headings) > 0: + hint = f'Correct solution: {"".join(row[field] for field in hint_headings if not pd.isna(row[field]))}' + else: + hint = f'Correct solution: {"".join(row[field] for field in golden_headings if not pd.isna(row[field]))}' + + tasks.append(Task( + input_values={field[len('INPUT:'):]: row[field] for field in input_headings if not pd.isna(row[field])}, + known_solutions=known_solutions, + message_on_unknown_solution=hint, + pool_id=pool_id)) + + return tasks + + +def create_tasks(toloka_client: TolokaClient, + pool_id: str, + pool_tasks_path: Optional[str] = None, + control_tasks_path: Optional[str] = None, + training_tasks_path: Optional[str] = None, + open_pool: Optional[bool] = False): + tasks: List[Task] = [] + if pool_tasks_path: + pool_tasks_df = pd.read_csv(pool_tasks_path) + tasks.extend(init_pool_tasks(pool_tasks_df, pool_id)) + if control_tasks_path: + control_tasks_df = pd.read_csv(control_tasks_path) + tasks.extend(init_control_tasks(control_tasks_df, pool_id)) + if training_tasks_path: + training_tasks_df = pd.read_csv(training_tasks_path) + tasks.extend(init_training_tasks(training_tasks_df, pool_id)) + + toloka_client.create_tasks(tasks, allow_defaults=True, open_pool=open_pool) + + +if __name__ == '__main__': + # command line arguments + parser = argparse.ArgumentParser(description='Create tasks for pool or training from csv file.') + parser.add_argument('--token_env', type=str, required=True, help='Environment variable for Toloka API access token') + parser.add_argument('--pool_id_path', type=str, required=True, help='Path to file with pool id') + parser.add_argument('--output_pool_id_path', type=str, required=True, + help='Path to store pool id with tasks created') + parser.add_argument('--tasks_csv', type=str, help='csv file with data for tasks') + parser.add_argument('--control_csv', type=str, help='csv file with data for control tasks') + parser.add_argument('--training_csv', type=str, help='csv file with data for training') + parser.add_argument('--open_pool', default=False, action="store_true", + help='Whether to open the pool when tasks created') + args = parser.parse_args() + + if not (args.tasks_csv or args.control_csv or args.training_csv): + parser.error('No csv files provided. ' + 'Please, add .csv file with --tasks_csv or --control_csv or --training_csv') + + toloka_token = os.environ[args.token_env] + + with open(args.pool_id_path) as pool_id_file: + pool_id = pool_id_file.readline().strip() + + toloka_client = TolokaClient(toloka_token, 'PRODUCTION') + create_tasks(toloka_client, pool_id, pool_tasks_path=args.tasks_csv, control_tasks_path=args.control_csv, + training_tasks_path=args.training_csv, open_pool=args.open_pool) + + with open(args.output_pool_id_path, 'w') as output_pool_id_file: + output_pool_id_file.write(pool_id) diff --git a/example/src/create_training.py b/example/src/create_training.py new file mode 100644 index 0000000..520a7f1 --- /dev/null +++ b/example/src/create_training.py @@ -0,0 +1,33 @@ +import argparse +import os + +from toloka.client import TolokaClient, Training + + +def create_training(toloka_client: TolokaClient, training_config_path: str, project_id: str) -> Training: + with open(training_config_path) as training_config_file: + json_string = training_config_file.read() + training = Training.from_json(json_string) + training.project_id = project_id + return toloka_client.create_training(training) + + +if __name__ == '__main__': + # command line arguments + parser = argparse.ArgumentParser(description='Create training in Toloka project') + parser.add_argument('--token_env', type=str, required=True, help='Environment variable for Toloka API access token') + parser.add_argument('--project_id_path', type=str, required=True, help='Path to file with project id') + parser.add_argument('--training_config_path', type=str, required=True, help='Path to a training config file') + parser.add_argument('--training_id_path', type=str, required=True, help='Path to store created training id') + args = parser.parse_args() + + toloka_token = os.environ[args.token_env] + + with open(args.project_id_path) as project_id_file: + project_id = project_id_file.readline().strip() + + toloka_client = TolokaClient(toloka_token, 'PRODUCTION') + training = create_training(toloka_client, args.training_config_path, project_id) + + with open(args.training_id_path, 'w') as training_id_file: + training_id_file.write(training.id) diff --git a/example/src/train_test_model.py b/example/src/train_test_model.py new file mode 100644 index 0000000..84e6665 --- /dev/null +++ b/example/src/train_test_model.py @@ -0,0 +1,82 @@ +import argparse +import json +import pickle +from typing import Optional, Tuple + +import numpy as np +import pandas as pd +from nltk.corpus import stopwords +from scipy.sparse import csr_matrix +from sklearn.base import ClassifierMixin +from sklearn.ensemble import RandomForestClassifier +from sklearn.feature_extraction.text import TfidfVectorizer +from sklearn.metrics import accuracy_score, f1_score + + +def get_X_y(data: pd.DataFrame) -> Tuple[pd.DataFrame, np.ndarray]: + data['category'] = data['category'].apply(lambda x: 1 if x == 'clickbait' else 0) + X, y = data.drop('category', axis=1), data['category'].to_numpy() + return X, y + + +def tfidf_preprocess(X_train: pd.DataFrame, X_test: Optional[pd.DataFrame] = None) -> Tuple[csr_matrix, csr_matrix]: + stop_words = set(stopwords.words("english")) + tfidf = TfidfVectorizer(stop_words=stop_words, ngram_range=(1, 2)) + tfidf_X_train = tfidf.fit_transform(X_train) + tfidf_X_test = tfidf.transform(X_test) + return tfidf_X_train, tfidf_X_test + + +def read_prepare_data(train_data: pd.DataFrame, + test_data: pd.DataFrame) -> Tuple[csr_matrix, csr_matrix, np.ndarray, np.ndarray]: + X_train, y_train = get_X_y(train_data) + X_test, y_test = get_X_y(test_data) + X_train, X_test = tfidf_preprocess(X_train.headline, X_test.headline) + return X_train, X_test, y_train, y_test + + +def train_model(X_train: csr_matrix, y_train: np.ndarray) -> ClassifierMixin: + classifier = RandomForestClassifier(class_weight='balanced') + classifier.fit(X_train, y_train) + return classifier + + +def test_model(classifier: ClassifierMixin, X_test: csr_matrix, y_test: np.ndarray) -> Tuple[float, float]: + test_preds = classifier.predict(X_test) + accuracy = accuracy_score(y_test, test_preds) + f1 = f1_score(y_test, test_preds) + return accuracy, f1 + + +def write_result(output_file_path: str, accuracy: float, f1: float): + with open(output_file_path, 'w') as output_file: + json.dump({'accuracy': accuracy, 'f1_score': f1}, output_file) + + +def save_model(classifier: ClassifierMixin, model_path: str): + with open(model_path, 'wb') as f: + pickle.dump(classifier, f) + + +if __name__ == '__main__': + # command line arguments + parser = argparse.ArgumentParser(description='Concatenate two datasets.') + parser.add_argument('--train_data', nargs='+', type=str, required=True, + help='Path to datasets. Concats into one if multiple') + parser.add_argument('--test_data', nargs='+', type=str, required=True, + help='Path to a test dataset. Concats into one if multiple') + parser.add_argument('--results_path', type=str, required=True, help='Path to store metrics on test dataset') + parser.add_argument('--model_path', type=str, help='Path to store trained model') + args = parser.parse_args() + + train_data = pd.concat([pd.read_csv(path) for path in args.train_data]) + test_data = pd.concat([pd.read_csv(path) for path in args.train_data]) + + X_train, X_test, y_train, y_test = read_prepare_data(train_data, test_data) + + classifier = train_model(X_train, y_train) + if args.model_path: + save_model(classifier, args.model_path) + + accuracy, f1 = test_model(classifier, X_test, y_test) + write_result(args.results_path, accuracy, f1) diff --git a/example/src/wait_pool.py b/example/src/wait_pool.py new file mode 100644 index 0000000..c5266f1 --- /dev/null +++ b/example/src/wait_pool.py @@ -0,0 +1,42 @@ +import argparse +import logging +import os.path +import time +from datetime import timedelta + +from toloka.client import TolokaClient +from toloka.client.analytics_request import CompletionPercentagePoolAnalytics + + +def wait_pool(toloka_token: str, pool_id: str, open_pool: bool = True, period: timedelta = timedelta(seconds=60)): + toloka_client = TolokaClient(toloka_token, 'PRODUCTION') + pool = toloka_client.get_pool(pool_id) + if pool.is_closed() and open_pool: + pool = toloka_client.open_pool(pool_id) + + while pool.is_open(): + op = toloka_client.get_analytics([CompletionPercentagePoolAnalytics(subject_id=pool_id)]) + percentage = toloka_client.wait_operation(op).details['value'][0]['result']['value'] + logging.info(f'Pool {pool_id} - {percentage}%') + + time.sleep(period.total_seconds()) + pool = toloka_client.get_pool(pool_id) + + +if __name__ == '__main__': + # command line arguments + parser = argparse.ArgumentParser(description='Open a pool on Toloka for annotating and wait for pool to complete.') + parser.add_argument('--token_env', type=str, required=True, help='Environment variable for Toloka API access token') + parser.add_argument('--pool_id_path', type=str, required=True, help='Path to file with pool id') + parser.add_argument('--output_pool_id_path', type=str, required=True, help='Path to store pool id with tasks created') + args = parser.parse_args() + + toloka_token = os.environ[args.token_env] + + with open(args.pool_id_path) as pool_id_file: + pool_id = pool_id_file.readline().strip() + + wait_pool(toloka_token, pool_id) + + with open(args.output_pool_id_path, 'w') as output_pool_id_file: + output_pool_id_file.write(pool_id) diff --git a/example/toloka_aggregate_assignments.json b/example/toloka_aggregate_assignments.json new file mode 100644 index 0000000..8910d88 --- /dev/null +++ b/example/toloka_aggregate_assignments.json @@ -0,0 +1,32 @@ +{ + "pipeline": { + "name": "toloka_aggregate_assignments" + }, + "description": "A pipeline for assignments aggregation from Toloka pool.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/aggregate_assignments.py", + "--token_env", "TOLOKA_API_ACCESS_KEY", + "--pool_id_path", "/pfs/toloka_wait_pool/pool.txt", + "--aggregated_tasks_path", "/pfs/out/results.csv" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "pfs": { + "repo": "toloka_wait_pool", + "glob": "/" + } + } +} diff --git a/example/toloka_create_control_tasks.json b/example/toloka_create_control_tasks.json new file mode 100644 index 0000000..5daa92f --- /dev/null +++ b/example/toloka_create_control_tasks.json @@ -0,0 +1,45 @@ +{ + "pipeline": { + "name": "toloka_create_control_tasks" + }, + "description": "A pipeline that creates control tasks for pool on Toloka.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/create_tasks_from_csv.py", + "--token_env", "TOLOKA_API_ACCESS_KEY", + "--pool_id_path", "/pfs/toloka_create_pool/pool.txt", + "--output_pool_id_path","/pfs/out/pool.txt", + "--control_csv", "/pfs/toloka_pool/control_tasks.csv" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "join": [ + { + "pfs": { + "repo": "toloka_pool", + "glob": "/", + "join_on": "$1" + } + }, + { + "pfs": { + "repo": "toloka_create_pool", + "glob": "/", + "join_on": "$1" + } + } + ] + } +} diff --git a/example/toloka_create_pool.json b/example/toloka_create_pool.json new file mode 100644 index 0000000..bb1f414 --- /dev/null +++ b/example/toloka_create_pool.json @@ -0,0 +1,53 @@ +{ + "pipeline": { + "name": "toloka_create_pool" + }, + "description": "A pipeline that creates pool for project on Toloka.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/create_pool.py", + "--token_env", "TOLOKA_API_ACCESS_KEY", + "--project_id_path", "/pfs/toloka_create_project/project.txt", + "--training_id_path", "/pfs/toloka_create_training/training.txt", + "--pool_config_path", "/pfs/toloka_pool/pool.json", + "--pool_id_path", "/pfs/out/pool.txt" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "join": [ + { + "pfs": { + "repo": "toloka_pool", + "glob": "/", + "join_on": "$1" + } + }, + { + "pfs": { + "repo": "toloka_create_project", + "glob": "/", + "join_on": "$1" + } + }, + { + "pfs": { + "repo": "toloka_create_training", + "glob": "/", + "join_on": "$1" + } + } + ] + } +} diff --git a/example/toloka_create_pool_tasks.json b/example/toloka_create_pool_tasks.json new file mode 100644 index 0000000..053ea58 --- /dev/null +++ b/example/toloka_create_pool_tasks.json @@ -0,0 +1,45 @@ +{ + "pipeline": { + "name": "toloka_create_pool_tasks" + }, + "description": "A pipeline that creates tasks for pool on Toloka.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/create_tasks_from_csv.py", + "--token_env", "TOLOKA_API_ACCESS_KEY", + "--pool_id_path", "/pfs/toloka_create_pool/pool.txt", + "--output_pool_id_path", "/pfs/out/pool.txt", + "--tasks_csv", "/pfs/toloka_pool/pool_tasks.csv" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "join": [ + { + "pfs": { + "repo": "toloka_pool", + "glob": "/", + "join_on": "$1" + } + }, + { + "pfs": { + "repo": "toloka_create_pool", + "glob": "/", + "join_on": "$1" + } + } + ] + } +} diff --git a/example/toloka_create_project.json b/example/toloka_create_project.json new file mode 100644 index 0000000..6bdfbb6 --- /dev/null +++ b/example/toloka_create_project.json @@ -0,0 +1,32 @@ +{ + "pipeline": { + "name": "toloka_create_project" + }, + "description": "A pipeline that creates project on Toloka.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/create_project.py", + "--token_env", "TOLOKA_API_ACCESS_KEY", + "--project_config_path", "/pfs/toloka_project_config/project.json", + "--project_id_path", "/pfs/out/project.txt" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "pfs": { + "repo": "toloka_project_config", + "glob": "/" + } + } +} diff --git a/example/toloka_create_training.json b/example/toloka_create_training.json new file mode 100644 index 0000000..60e6a93 --- /dev/null +++ b/example/toloka_create_training.json @@ -0,0 +1,46 @@ +{ + "pipeline": { + "name": "toloka_create_training" + }, + "description": "A pipeline that creates training for project on Toloka.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/create_training.py", + "--token_env", + "TOLOKA_API_ACCESS_KEY", + "--project_id_path", "/pfs/toloka_create_project/project.txt", + "--training_config_path", "/pfs/toloka_training_config/training.json", + "--training_id_path", "/pfs/out/training.txt" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "join": [ + { + "pfs": { + "repo": "toloka_training_config", + "glob": "/", + "join_on": "$1" + } + }, + { + "pfs": { + "repo": "toloka_create_project", + "glob": "/", + "join_on": "$1" + } + } + ] + } +} diff --git a/example/toloka_create_training_tasks.json b/example/toloka_create_training_tasks.json new file mode 100644 index 0000000..ffee25e --- /dev/null +++ b/example/toloka_create_training_tasks.json @@ -0,0 +1,46 @@ +{ + "pipeline": { + "name": "toloka_create_training_tasks" + }, + "description": "A pipeline that creates training tasks for project on Toloka.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/create_tasks_from_csv.py", + "--token_env", "TOLOKA_API_ACCESS_KEY", + "--pool_id_path", "/pfs/toloka_create_training/training.txt", + "--output_pool_id_path", "/pfs/out/training.txt", + "--training_csv", "/pfs/toloka_training_tasks/training_tasks.csv", + "--open_pool" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "join": [ + { + "pfs": { + "repo": "toloka_training_tasks", + "glob": "/", + "join_on": "$1" + } + }, + { + "pfs": { + "repo": "toloka_create_training", + "glob": "/", + "join_on": "$1" + } + } + ] + } +} diff --git a/example/toloka_wait_pool.json b/example/toloka_wait_pool.json new file mode 100644 index 0000000..456c48a --- /dev/null +++ b/example/toloka_wait_pool.json @@ -0,0 +1,32 @@ +{ + "pipeline": { + "name": "toloka_wait_pool" + }, + "description": "A pipeline that opens a pool for annotation creates project on Toloka.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/wait_pool.py", + "--token_env", "TOLOKA_API_ACCESS_KEY", + "--pool_id_path", "/pfs/toloka_create_pool_tasks/pool.txt", + "--output_pool_id_path", "/pfs/out/pool.txt" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "pfs": { + "repo": "toloka_create_pool_tasks", + "glob": "/" + } + } +} diff --git a/example/train_test_model.json b/example/train_test_model.json new file mode 100644 index 0000000..9ded589 --- /dev/null +++ b/example/train_test_model.json @@ -0,0 +1,45 @@ +{ + "pipeline": { + "name": "train_test_model" + }, + "description": "A pipeline that trains Random Forest classifier and evaluate Accuracy and F1 score on test data.", + "transform": { + "image": "toloka_pachyderm:latest", + "cmd": [ + "python3", + "/code/train_test_model.py", + "--train_data", "/pfs/clickbait_data/train.csv", "/pfs/toloka_aggregate_assignments/results.csv", + "--test_data", "/pfs/clickbait_data/test.csv", + "--results_path", "/pfs/out/random_forest_test.json", + "--model_path", "/pfs/out/rf_model.pkl" + ], + "secrets": [ + { + "name": "toloka-api", + "env_var": "TOLOKA_API_ACCESS_KEY", + "key": "token" + } + ] + }, + "parallelism_spec": { + "constant": "1" + }, + "input": { + "join": [ + { + "pfs": { + "repo": "clickbait_data", + "glob": "/", + "join_on": "$1" + } + }, + { + "pfs": { + "repo": "toloka_aggregate_assignments", + "glob": "/", + "join_on": "$1" + } + } + ] + } +}