Skip to content

Commit

Permalink
Memory leak patch (#9)
Browse files Browse the repository at this point in the history
* patch.

* update package ver.

* delete unwanted files.

* memory leak debug, import debug, range-loop-consturc debug

* add testcase.

* minor change.

* minor change.

* minor change.

* minor change.

* doc update.
  • Loading branch information
Danial-Alh authored Nov 18, 2021
1 parent d8726ae commit e4c4340
Showing 7 changed files with 23 additions and 14 deletions.
8 changes: 7 additions & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -9,7 +9,13 @@ jobs:
- name: Checkout code
uses: actions/checkout@v2

- name: Build package
- name: Install packages
run: pip install -r requirements.txt

- name: Run test case
run: python test_cases.py

- name: Setup package
run: python setup.py sdist

- name: Publish a Python distribution to PyPI
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# fast-bleu Package

This is a fast multithreaded C++ implementation of NLTK BLEU with Python wrapper; computing BLEU and SelfBLEU score for a fixed reference set.
It can return (Self)BLEU for different (max) n-grams simultaneously and efficiently (e.g. BLEU-2, BLEU-3 and etc.).
This is a fast multithreaded C++ implementation of NLTK BLEU with Python wrapper; computing BLEU and SelfBLEU scores for a fixed reference set.
It can return (Self)BLEU for different (max) n-grams simultaneously and efficiently (e.g. BLEU-2, BLEU-3, etc.).

## Installation

@@ -18,10 +18,10 @@ pip install --user fast-bleu

### MacOS

As the macOS uses clang and it does not support OpenMP; one workaround is to first install gcc with `brew install gcc`. After that, gcc specific binaries will be added (for example, it will be maybe `gcc-10`
As the macOS uses clang and it does not support OpenMP, one workaround is to first install gcc with `brew install gcc`. After that, gcc specific binaries will be added (for example, it will be maybe `gcc-10`
and `g++-10`).

To change the default compiler, an option to the installation command is added. So you can install the [PyPI latest stable release](https://pypi.org/project/fast-bleu/) with the following command:
To change the default compiler, an option to the installation command is added, so you can install the [PyPI latest stable release](https://pypi.org/project/fast-bleu/) with the following command:

``` bash
pip install --user fast-bleu --install-option="--CC=<path-to-gcc>" --install-option="--CXX=<path-to-g++>"
4 changes: 2 additions & 2 deletions fast_bleu/__python_wrapper__.py
Original file line number Diff line number Diff line change
@@ -127,7 +127,7 @@ def get_score(self, hypotheses: list):
return {self.__weight_keys[i]: r for i, r in enumerate(result)}

def __del__(self):
if hasattr(self, '__instance') and hasattr(self, '__del_instance '):
if hasattr(self, '_BLEU__instance') and hasattr(self, '_BLEU__del_instance'):
self.__del_instance(self.__instance)


@@ -240,5 +240,5 @@ def get_score(self):
return {self.__weight_keys[i]: r for i, r in enumerate(result)}

def __del__(self):
if hasattr(self, '__instance') and hasattr(self, '__del_instance '):
if hasattr(self, '_SelfBLEU__instance') and hasattr(self, '_SelfBLEU__del_instance'):
self.__del_instance(self.__instance)
5 changes: 3 additions & 2 deletions fast_bleu/cpp_sources/sources/nltk.cpp
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
#include <limits>
#include <vector>
#include <string>
#include <cmath>
@@ -60,11 +61,11 @@ Fraction modified_precision(CustomMap **reference_max_counts,
int numerator = 0;
int denominator = 0;

for (pair<string, int> const &p : counts)
for (auto const &p : counts)
denominator += counts.get(p.first);
denominator = max(1, denominator);

for (pair<string, int> const &p : counts)
for (auto const &p : counts)
numerator += min(counts.get(p.first), max_counts.get(p.first));

delete hyp_ngrams;
4 changes: 2 additions & 2 deletions fast_bleu/cpp_sources/sources/self_bleu.cpp
Original file line number Diff line number Diff line change
@@ -227,7 +227,7 @@ vector<vector<double>> SELF_BLEU_CPP::get_score()
refs[s + t] = references[s];
}
for (int n = 0; n < curr_n; n++)
for (pair<string, int> const &p : *(references_counts[n][i]))
for (auto const &p : *(references_counts[n][i]))
{
string const &ng = p.first;
if (reference_max_counts[n]->get(ng) == references_counts[n][i]->get(ng))
@@ -243,7 +243,7 @@ vector<vector<double>> SELF_BLEU_CPP::get_score()
smoothing_function,
auto_reweight);
for (int n = 0; n < curr_n; n++)
for (pair<string, int> const &p : *(references_counts[n][i]))
for (auto const &p : *(references_counts[n][i]))
{
string const &ng = p.first;
(*ref_max_counts[n])[ng] = reference_max_counts[n]->get(ng);
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
@@ -70,7 +70,7 @@ def get_ext_filename(self, ext_name):

setup = setuptools.setup(
name='fast-bleu',
version="0.0.89",
version="0.0.90",
author="Danial Alihosseini",
author_email="danial.alihosseini@gmail.com",
description="A fast multithreaded C++ implementation of nltk BLEU with python wrapper.",
6 changes: 4 additions & 2 deletions test_cases.py
Original file line number Diff line number Diff line change
@@ -29,7 +29,9 @@ def nltk_bleu(refs, hyps):
def cpp_bleu(refs, hyps):
w = {i: list(np.ones(i) / (i)) for i in range(2, 6)}
bleu = BLEU(refs, w, verbose=True)
return bleu.get_score(hyps)[max_n]
score = bleu.get_score(hyps)[max_n]
del bleu
return score


def nltk_self_bleu(refs, hyps):
@@ -93,7 +95,7 @@ def compare(nltk_func, cpp_func):

compare(nltk_org_bleu, cpp_bleu)
# compare(nltk_bleu, cpp_bleu)
# compare(nltk_self_bleu, cpp_self_bleu)
compare(nltk_self_bleu, cpp_self_bleu)

# res, ti = get_execution_time(cpp_bleu)
# res, ti = get_execution_time(cpp_self_bleu)

0 comments on commit e4c4340

Please sign in to comment.