module util::Sampling
Utilities to randomly select smaller datasets from larger datasets
Usage
import util::Sampling;
Dependencies
import util::Math;
import Map;
import List;
import Set;
Description
Sampling is important when the analysis algorithms do not scale to the size of the original corpus, or when you need to train an analysis on a representative set without overfitting on the entire corpus. These sampling functions all assume that a uniformly random selection is required.
function sample
Reduce the arity of a set by selecting a uniformly distributed sample.
set[&T] sample(set[&T] corpus, int target)
A uniform subset is computed by iterating over the set and skipping every element
with a probability of 1/(size(corpus) / target). This rapidly generates a new set of
expected target size, but most probably a little smaller or larger.
Examples
rascal>import util::Sampling;
ok
rascal>sample({"a","b","c","e","f","g","h","i","j","k"}, 4)
set[str]: {"a","b","c","e","f","g","h","i","j","k"}
rascal>sample({"a","b","c","e","f","g","h","i","j","k"}, 4)
set[str]: {"b","e","f","h","i","k"}
rascal>sample({"a","b","c","e","f","g","h","i","j","k"}, 4)
set[str]: {"b","c","e","j","k"}
function sample
Reduce the length of a list by selecting a uniformly distributed sample.
list[&T] sample(list[&T] corpus, int target)
The random selection of elements does not change their initial order in the list.
A uniform sublist is computed by iterating over the list and skipping every element
with a probability of 1/(size(corpus) / target). This rapidly generates a new list of
expected target size, but most probably a little smaller or larger.
Examples
rascal>import util::Sampling;
ok
rascal>sample([1..1000], 30)
list[int]: [3,49,66,127,166,200,206,243,288,348,370,384,454,471,493,526,555,557,578,592,599,655,682,700,703,775,836,843,845,878,884,885,910,954]
rascal>sample([1..1000], 30)
list[int]: [34,41,126,139,200,229,286,292,375,388,487,489,516,520,583,599,644,668,707,730,734,771,776,846,852,869,932,947,964,967,975]
rascal>sample([1..1000], 30)
list[int]: [29,35,84,111,123,162,173,191,206,234,237,264,268,303,324,362,389,403,424,430,431,469,494,497,532,561,587,644,654,670,716,721,729,746,789,822,876,892,999]
function sample
Reduce the size of a map by selecting a uniformly distributed sample.
map[&T,&U] sample(map[&T,&U] corpus, int target)
A uniform submap is computed by iterating over the map's keys and skipping every key
with a probability of 1/(size(corpus) / target). This rapidly generates a new map of
expected target size, but most probably a little smaller or larger.