What will we miss out of ElasticSearch? Everything since we use Hive.
ElasticSearch | Hive | Moor+FFI |
|---|---|---|
Elasticsearch has index which is a collection of documents that are related to each other | No index and relationship | Has index and relationship |
inverted index, which is designed to allow very fast full-text searches by creating list of unique words of the document | No full text search | SQL index, query and join |
Sorting ability Sort.by relevance or custom fields | No sorting ability | SQL order by |
Paginator service for complex Pagination | No Pagination | No pagination |
How much data would we be storing ?
As requested per Pots is limited to 500 Transaction History which is in average equivalent to 1 year data.
How does the search behave?
A pot is clicked, then all 500 transaction history is loaded into memory but 20 items are displayed in a page.
A single string will search on all 4 parameters with all its values uppercased: Date, partner, category, and description.
Current approach
Why not Moor+FFI since its closer to ElasticSearch? Based on discussion with other TLs, since only Transaction History uses this search; We decided to still use Hive and build searching ability on top of Hive.
Very specific only to Transaction History, the searching capability is customized
Development relies on Dart collection filtering and sorting abilities
Thanks to Rashmi Ranganathan (Unlicensed) and Zainudin Saputra (Unlicensed) we will make this generic in the next development Sprint.
below is the detail:
class TransactionHistoryQueryUtil {
static ListTransactionHistoryModel filter(
{List<TransactionHistoryModel> transactions,
double fromAmount,
double toAmount,
String fromDate,
String toDate,
String searchText,
List<String> categoryCodesList,
int page,
int limit,
bool isSorted = false,
bool isPaginated = false}) {
if (fromAmount > 0 && toAmount > 0) {
transactions = transactions
.where((t) => fromAmount <= t.amount && t.amount <= toAmount)
.toList();
}
if (fromDate != null && toDate != null) {
transactions = transactions
.where((t) =>
DateTime.parse(fromDate).millisecondsSinceEpoch <=
DateTime.parse(t.transactionDate).millisecondsSinceEpoch &&
DateTime.parse(t.transactionDate).millisecondsSinceEpoch <=
DateTime.parse(toDate).millisecondsSinceEpoch)
.toList();
}
if (searchText != null) {
searchText = searchText.toUpperCase();
transactions = transactions.where((t) {
final String formattedDateId =
DateFormat(DateTimeFormatConstants.ddMMMMyyyy, 'id')
.format(DateTime.parse(t.transactionDate));
final String formattedDateEn =
DateFormat(DateTimeFormatConstants.ddMMMMyyyy, 'en')
.format(DateTime.parse(t.transactionDate));
return t.amount.toString().toUpperCase().contains(searchText) ||
t.partnerName.bahasa.toUpperCase().contains(searchText) ||
t.partnerName.english.toUpperCase().contains(searchText) ||
t.category.bahasa.toUpperCase().contains(searchText) ||
t.category.english.toUpperCase().contains(searchText) ||
formattedDateId.toUpperCase().contains(searchText) ||
formattedDateEn.toUpperCase().contains(searchText);
}).toList();
}
if (categoryCodesList != null) {
if (categoryCodesList.isNotEmpty) {
categoryCodesList =
categoryCodesList.map((c) => c.replaceAll('\"', '')).toList();
transactions = transactions
.where((t) => categoryCodesList.contains(t.category.code))
.toList();
}
}
if (isSorted) {
transactions = transactions
..sort((t1, t2) {
return DateTime.parse(t2.transactionDate)
.millisecondsSinceEpoch
.compareTo(
DateTime.parse(t1.transactionDate).millisecondsSinceEpoch);
});
}
if (isPaginated) {
int lastIndex = page * limit;
if (lastIndex > transactions.length) {
lastIndex = transactions.length;
}
final transactionResultsPaged =
transactions.sublist((page - 1) * limit, lastIndex).toList();
return ListTransactionHistoryModel(
total: transactions.length,
limit: limit,
page: page,
data: transactionResultsPaged);
}
return ListTransactionHistoryModel(
total: transactions.length,
limit: limit,
page: page,
data: transactions);
}
}
Next Step
We were trying to make the search generic at local datasource instead repository level. We were facing some issues. For this new project, we can actually implement that. We should try to make a common filter mechanism. cc Martin Vodila (Unlicensed)