...
Quite often, our customers are faced with the necessity to download information about calls from the Call Detail Record section to themselvestheir systems. As you already know, all historical statistics are stored in the Elasticsearch database. In the documentation, you could also see an example of receiving data using our REST API. But, what if the calls there are several hundred? Or a few thousandhundreds or thousands of calls? How to get this amount of data? Today I will tell you how to work with large amounts of data This article will answer this question.
Scroll
To get a lot of data, Elasticsearch provides the scroll function, which we repeated in our REST API. Consider an An example:
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
{
"scroll" : "5m",
"limit": 1000,
"sort": {
"created_time": {
"order": "desc",
"unmapped_type": "boolean"
}
},
"index": "cdr-a",
"query": "*",
"columns": [
"created_time",
"uuid",
"direction",
"duration"
],
"filter": [
{
"bool": {
"must": [
{
"range": {
"created_time": {
"gte": "now/w",
"lte": "now"
}
}
}
]
}
}
]
} |
...
We repeat the request until we collect all the data from the server.
Bonus
As a bonus, I we have prepared a small bash script, which is using cURL and jq will help you download the necessary data and save it in a CSV file:
...