1[![Downloads](https://pepy.tech/badge/benford-py)](https://pepy.tech/project/benford-py) 2 3# Benford for Python 4 5-------------------------------------------------------------------------------- 6 7**Citing** 8 9 10If you find *Benford_py* useful in your research, please consider adding the following citation: 11 12```bibtex 13@misc{benford_py, 14 author = {Marcel, Milcent}, 15 title = {{Benford_py: a Python Implementation of Benford's Law Tests}}, 16 year = {2017}, 17 publisher = {GitHub}, 18 journal = {GitHub repository}, 19 howpublished = {\url{https://github.com/milcent/benford_py}}, 20} 21``` 22 23-------------------------------------------------------------------------------- 24 25`current version = 0.5.0` 26 27### See [release notes](https://github.com/milcent/benford_py/releases/) for features in this and in older versions 28 29### Python versions >= 3.6 30 31### Installation 32 33Benford_py is a package in PyPi, so you can install with pip: 34 35`pip install benford_py` 36 37or 38 39`pip install benford-py` 40 41Or you can cd into the site-packages subfolder of your python distribution (or environment) and git clone from there: 42 43`git clone https://github.com/milcent/benford_py` 44 45For a quick start, please go to the [Demo notebook](https://github.com/milcent/benford_py/blob/master/Demo.ipynb), in which I show examples on how to run the tests with the SPY (S&P 500 ETF) daily returns. 46 47For more fine-grained details of the functions and classes, see the [docs](https://benford-py.readthedocs.io/en/latest/index.html). 48 49### Background 50 51The first digit of a number is [its leftmost digit](https://github.com/milcent/benford_py/blob/master/img/First_Digits.png) 52 53Since the first digit of any number can range from "1" to "9" 54(not considering "0"), it would be intuitively expected that the 55proportion of each occurrence in a set of numerical records would 56be uniformly distributed at 1/9, i.e., approximately 0.1111, 57or 11.11%. 58 59[Benford's Law](https://en.wikipedia.org/wiki/Benford%27s_law), 60also known as the Law of First Digits or the Phenomenon of 61Significant Digits, is the finding that the first digits of the 62numbers found in series of records of the most varied sources do 63not display a uniform distribution, but rather are arranged in such 64a way that the digit "1" is the most frequent, followed by "2", 65"3", and so in a successive and decremental way down to "9", 66which presents the lowest frequency as the first digit. 67 68The expected distributions of the First Digits in a 69Benford-compliant data set are the ones shown [here](https://github.com/milcent/benford_py/blob/master/img/First.png) 70 71The first record on the subject dates from 1881, in the work of 72[Simon Newcomb](https://github.com/milcent/benford_py/blob/master/img/Simon_Newcomb_APS.jpg), an American-Canadian astronomer and mathematician, 73who noted that in the logarithmic tables the first pages, which 74contained logarithms beginning with the numerals "1" and "2", 75were more worn out, that is, more consulted. 76 77In that same article, Newcomb proposed the [formula](https://github.com/milcent/benford_py/blob/master/img/formula.png) for the probability of a certain digit "d" 78being the first digit of a number, given by the following equation. 79 80In 1938, the American physicist [Frank Benford](https://github.com/milcent/benford_py/blob/master/img/2429_Benford-Frank.jpg) revisited the 81phenomenon, which he called the "Law of Anomalous Numbers," in 82a survey with more than 20,000 observations of empirical data 83compiled from various sources, ranging from areas of rivers to 84molecular weights of chemical compounds, including cost data, 85address numbers, population sizes and physical constants. All 86of them, to a greater or lesser extent, followed such 87distribution. 88 89The extent of Benford's work seems to have been one good reason 90for the phenomenon to be popularized with his name, though 91described by Newcomb 57 years earlier. 92 93Derivations of the original formula were also applied in the 94expected findings of the proportions of digits in other 95positions in the number, as in the case of the second digit 96(BENFORD, 1938), as well as combinations, such as the first 97two digits of a number (NIGRINI, 2012, p.5). 98 99Only in 1995, however, was the phenomenon proven by Hill. 100His proof was based on the fact that numbers in data series 101following the Benford Law are, in effect, "second generation" 102distributions, ie combinations of other distributions. 103The union of randomly drawn samples from various distributions 104forms a distribution that respects Benford's Law (HILL, 1995). 105 106When grouped in ascending order, data that obey Benford's Law 107must approximate a geometric sequence (NIGRINI, 2012, page 21). 108From this it follows that the logarithms of this ordered series 109must form a straight line. In addition, the mantissas (decimal 110parts) of the logarithms of these numbers must be uniformly 111distributed in the interval [0,1] (NIGRINI, 2012, p.10). 112 113In general, a series of numerical records follows Benford's Law 114when (NIGRINI, 2012, p.21): 115* it represents magnitudes of events or events, such as populations 116of cities, flows of water in rivers or sizes of celestial bodies; 117* it does not have pre-established minimum or maximum limits; 118* it is not made up of numbers used as identifiers, such as 119identity or social security numbers, bank accounts, telephone numbers; and 120* its mean is less than the median, and the data is not 121concentrated around the mean. 122 123It follows from this expected distribution that, if the set of 124numbers in a series of records that usually respects the Law 125shows a deviation in the proportions found, there may be 126distortions, whether intentional or not. 127 128Benford's Law has been used in [several fields](http://www.benfordonline.net/). 129Afer asserting that the usual data type is Benford-compliant, 130one can study samples from the same data type tin search of 131inconsistencies, errors or even [fraud](https://www.amazon.com.br/Benfords-Law-Applications-Accounting-Detection/dp/1118152859). 132 133This open source module is an attempt to facilitate the 134performance of Benford's Law-related tests by people using 135Python, whether interactively or in an automated, scripting way. 136 137It uses the versatility of numpy and pandas, along with 138matplotlib for vizualization, to deliver results like [this one](https://github.com/milcent/benford_py/blob/master/img/SPY-f2d-conf_level-95.png) and much more. 139 140 141It has been a long time since I last tested it in Python 2. The death clock has stopped ticking, so officially it is for Python 3 now. It should work on Linux, Windows and Mac, but please file a bug report if you run into some trouble. 142 143Also, if you have some nice data set that we can run these tests on, let'us try it. 144 145Thanks! 146 147Milcent 148