Lazy Loading DataTable with IceFaces

IceFaces provides numerous components to facilitate the development of web applications. Among these, one of the most 'useful' is definitely ice:dataTable. This component, together with ice:dataPaginator makes it possible to paginate the entire data set of a table, showing only N rows per page. The main weakness of this component lies in the difficulty to handle large data sets: ice:dataTable component need to receive a list containing all rows that will gradually showed. As long as we are in the order of some hundreds of records, there is no problem to provide the entire results list to the component; but if they begin to be thousands, keep in memory such a quantity of objects can be an issue. The solution is to manage the list of results in a lazy way: the list itself will retrieve the records to show in the current page and only when there will be a real need. Besides this, we need also to manage the total page number: the paginator in fact invokes the method size() of the list supplied to the table calculates the number of pages. Let's see three possible implementations to implement the lazy loading of the list. They will be initialized in the page backingBean and used in this way in the .jsf template:


....


...

h2. 1. LazyLoadingList This list keeps in memory a list of objects corresponding to the first page. When the user navigates in the following pages, they will be retrieved by method get() and saved in a different list. Note that the constructor of the list accepts as incoming parameter totalResultsNumber, that is the total number of results to show in the table. This parameter 'is usually the result of a count query.

package com.devinterface.lazyloading;

import java.util.AbstractList;
import java.util.List;
pre
/**
 * This list loads and stores only the first page and the current page.
 * If pageSize is equals to totalResultsNumber, the dataTable will be non paginated: the first query will retrieve all the dataset.
 *
 * @param 
 */
public class LazyLoadingList extends AbstractList
{
  private IDataProvider dataProvider;

  private List firtsPageData;
  private List currentPageData;

  private int currentPage = -1;
  private int totalResultsNumber;
  private int pageSize;

  /**
   * @param dataProvider, the object that will perform the query
   * @param pageSize, the number of rows to be showed in a table page
   * @param totalResultsNumber, the total number of rows as result of the database count query. 
   */
  public LazyLoadingList(IDataProvider dataProvider, int pageSize, int totalResultsNumber)
  {
    this.dataProvider = dataProvider;
    this.totalResultsNumber = totalResultsNumber;
    this.pageSize = pageSize;
  }

  @Override
  public T get(int i)
  {
    if (i < pageSize)
    {
      if (firtsPageData == null)
        firtsPageData = dataProvider.getBufferedData(i, pageSize);
      return firtsPageData.get(i);
    }
    int page = i / pageSize;

    if (page != currentPage)
    {
      currentPage = page;
      currentPageData = dataProvider.getBufferedData(i, pageSize);
    }

    return currentPageData.get(i % pageSize);
  }

  @Override
  public int size()
  {
    return totalResultsNumber;
  }

  public void setTotalResultsNumber(int totalResultsNumber)
  {
    this.totalResultsNumber = totalResultsNumber;
  }

  @Override
  public void clear()
  {
    firtsPageData.clear();
    currentPageData.clear();
  }

}
h2. 2. LazyLoadingMapList This list keeps retrieved records in a HashMap. Note that the map is never cleared, so in the worst case (the user scrolls one by one all the pages) it will contains all elements of the dataset.

package com.devinterface.lazyloading;

import java.util.AbstractList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * This list stores all records in a map with key = "table row index" and value = "T"
 * If pageSize is equals to totalResultsNumber, the dataTable will be non paginated: the first query will retrieve all the dataset.
 * Note that if a user moves from first to last page, one page at time, all record will be in memory.
 *
 * @param 
 */
public class LazyLoadingMapList extends AbstractList
{
  private IDataProvider dataProvider;

  private int totalResultsNumber;
  private int pageSize;

  /** cache of loadedData items */
  private Map loadedData;

  /**
   * @param dataProvider, the object that will perform the query
   * @param pageSize, the number of rows to be considered as "a page"
   * @param totalResultsNumber, the total number of rows as result of the database count query. 
   */
  public LazyLoadingMapList(IDataProvider dataProvider, int pageSize, int totalResultsNumber)
  {
    this.dataProvider = dataProvider;
    this.totalResultsNumber = totalResultsNumber;
    this.pageSize = pageSize;
    loadedData = new HashMap();
  }

  @Override
  public T get(int i)
  {
    if (!loadedData.containsKey(i))
    {
      int pageIndex = i / pageSize;
      List results = dataProvider.getBufferedData(i, pageSize);
      for (int j = 0; j < results.size(); j++)
      {
        loadedData.put(Integer.valueOf(pageIndex * pageSize + j), (T) results.get(j));
      }
    }
    return loadedData.get(i);

  }

  @Override
  public int size()
  {
    return totalResultsNumber;
  }

  public void setTotalResultsNumber(int totalResultsNumber)
  {
    this.totalResultsNumber = totalResultsNumber;
  }

  @Override
  public void clear()
  {
    loadedData.clear();
  }

}
h2. 3. LazyLoadingBufferedMapList This list represents an evolution of the previous one, keeping in memory only the elements corresponding to the current page, the previous and the next.

package com.devinterface.lazyloading;

import java.util.AbstractList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * This list stores records in a map with key = "table row index" and value = "element"
 * It keeps a buffer of 3 pages: the first time it will load pages 1,2,3. 
 * When user moves to page 4, all stored data will be cleared and will be retrieved page 3,4,5.
 * Basically, this list will store current page, previous page and next page. 
 * If bufferSize is equals to totalResultsNumber, the dataTable will be non paginated: the first query will retrieve all the dataset.
 *
 * @param 
 */
public class LazyLoadingBufferedMapList extends AbstractList
{
  private IDataProvider dataAdapter;

  private int totalResultsNumber;
  private int pageSize = 10;
  private int bufferSize = 30;

  /** cache of loadedData items */
  private Map loadedData;

  /**
   * 
   * @param dataAdapter, the object that will perform the query
   * @param pageSize, the number of rows to be considered as "a page"
   * @param totalResultsNumber, the total number of rows as result of the database query. 
   */
  public LazyLoadingBufferedMapList(IDataProvider dataAdapter, int pageSize, int totalResultsNumber)
  {
    this.dataAdapter = dataAdapter;
    this.totalResultsNumber = totalResultsNumber;
    this.pageSize = pageSize;
    this.bufferSize = pageSize * 3;
    loadedData = new HashMap();
  }

  @Override
  public T get(int i)
  {
    if (!loadedData.containsKey(i))
    {
      clearMap();

      int startRow = getStartRow(i);

      int numElementToFind = bufferSize;
      if ((startRow + numElementToFind) > totalResultsNumber)
        numElementToFind = totalResultsNumber - startRow;

      List results = dataAdapter.getBufferedData(startRow, numElementToFind);
      for (int j = 0; j < results.size(); j++)
        loadedData.put((startRow + j), (T) results.get(j));
    }
    return loadedData.get(i);

  }

  /**
   * clears the map except the first element that MUST be kept
   */
  private void clearMap()
  {
    T firstElement = loadedData.get(0);
    loadedData.clear();
    loadedData.put(0, firstElement);
  }

  /**
   * Calculates the index of the previous page's first element
   * @param i, the current row index
   * @return the index of the previous page's first element
   */
  private int getStartRow(int i)
  {
    int currentPage = (i / pageSize) + 1;

    int firstIndexOfCurrentPage = pageSize * (currentPage - 1);

    int firstIndexOfPreviusPage = firstIndexOfCurrentPage - (bufferSize / 3);

    if (firstIndexOfPreviusPage < 0)
      firstIndexOfPreviusPage = 0;

    return firstIndexOfPreviusPage;
  }

  @Override
  public int size()
  {
    return totalResultsNumber;
  }

  public void setNumResults(int numResults)
  {
    this.totalResultsNumber = numResults;
  }

  @Override
  public void clear()
  {
    loadedData.clear();
  }
}
h2. Conclusions All implementations give the opportunity to retrieve records in a lazy way. Surely the second solution is the most potentially weak as in the worst case will keep in memory all the dataset. Surely the third implementation is the best solution from all points of view. It allows to have in memory only a limited number of records and to meet any potential "back and forth" of the user. Sources on my GitHub repository "LazyLoadingDataTable":http://github.com/xponrails/LazyLoadingDataTable